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PREFACE 


This  paper  summarizes  the  results  of  an  intensive  discussion 
with  Lt  Col  Ron  Dukes,  Chief  of  the  HQ  MAC  C-130  ATS  on-site 
liaison  group  (MACOS  OL  Q)  at  Little  Rock  AFB,  AR.  The  purpose  of 
the  discussion  was  to  document  the  experiences  and  "lessons 
learned"  by  Lt  Col  Dukes  during  the  development  and  implementation 
of  the  C-130  ATS.  This  information  was  collected  as  part  of  an 
ongoing  effort  to  develop  a  training  systems  "lessons  learned" 
database.  The  database  is  one  element  of  a  larger  program 
concerned  with  the  development  of  principles  and  guidelines  for  the 
design,  development,  implementation,  evaluation,  and  operation  of 
total  aircrew  training  systems  which  is  being  supported  by  the 
Aircrew  Training  Research  Division  of  the  Armstrong  Laboratory 
under  Contract  F33615-90-C-0005  with  the  University  of  Daytcn 
Research  Institute. 
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SUMMARY 


This  paper  documents  the  results  of  a  meeting  with  Lt  Col  Ron 
Dukes,  the  chief  of  the  HQ  MAC  operating  location  (MACOS  OL  Q)  with 
the  C-130  ATS  program  at  Little  Rock  AFB,  AR.  The  purpose  of  the 
meeting,  which  was  recorded  on  audiotape,  was  to  document  some  of 
Lt  Col  Dukes'  key  experiences  and  "lessons  learned"  during 
development  of  the  C-130  ATS.  The  information  in  this  paper  is  an 
edited  transcription  of  Lt  Col  Dukes'  remarks.  Thus,  it  should  be 
noted  that,  although  written  in  the  first  person,  much  of  the 
content  is  actually  paraphrased.  During  the  course  of  the  meeting, 
Lt  Col  Dukes  discussed  his  experiences  and  "lessons  learned"  in  a 
number  of  functional  areas  of  the  C-130  ATS  program  including: 
courseware,  training  management,  test  and  evaluation,  quality 
assurance,  and  configuration  management.  He  concluded  his 
presentation  with  a  number  of  generic  lessons  learned  which  provide 
a  profitable  source  of  guidance  for  all  military  organizations 
involved  in  the  acquisition  of  contractor  developed  and  operated 
training  systems. 


LESSONS  LEARNED  IN  THE  DEVELOPMENT  OF  THE  C-130  AIRCREW  TRAINING 
SYSTEM:  A  SUMMARY  OF  AIR  FORCE  ON-SITE  EXPERIENCE 

I .  INTRODUCTION 

This  paper  documents  the  results  of  a  meeting  with  Lt 
Col  Ron  Dukes,  the  former  chief  of  the  Headquarters 
Military  Airlift  Command  Operating  Location  (HQ  MAC  MACOS 
OL  Q)  with  the  C-130  Aircrew  Training  System  (ATS)  program 
at  Little  Rock  AFB,  AR.  The  purpose  of  the  meeting, 
which  was  recorded  on  audiotape,  was  to  document  some  of 
Lt  Col  Dukes'  extensive  experience  and  "lessons  learned" 
during  development  and  implementation  of  the  C-130  ATS. 
This  information  was  collected  as  part  of  an  effort  to 
develop  a  training  systems  "lessons  learned"  database.  The 
database  is  part  of  a  larger  program  concerned  with  the 
development  of  principles  and  guidelines  for  the  design, 
development,  implementation,  evaluation,  and  operation  of 
total  aircrew  training  systems  which  is  being  supported 
by  the  University  of  Dayton  Research  Institute  under  a 
contract  with  Aircrew  Training  Research  Division  of  the 
Armstrong  Laboratory  (Rockway  &  Nullmeyer,  1990) . 

Background 

The  current  trend  within  the  Air  Force  is  to  design  aircrew 
training  programs  as  total  integrated  systems  rather  than  as 
collections  of  courses  or  blocks  of  instruction.  This  trend  has 
been  coupled  with  a  concurrent  shift  to  contracting  out  the  design, 
delivery  and  support  of  aircrew  training  (Grossman,  1989) .  These 
changes  have  introduced  a  new  set  of  technical  and  management 
issues  which  impact  the  design,  development,  implementation, 
evaluation,  and  operation  of  aircrew  training  programs.  The 
Aircrew  Training  Research  Division  of  the  Armstrong  Laboratory  is 
conducting  R&D  to  address  several  of  these  issues  in  order  to 
provide  principles,  procedures,  and  user-oriented  guidelines  to 
support  Air  Force  acquisition  and  operational  training  agencies. 

Currently,  a  number  of  Air  Force  aircrew  training  programs  are 
being  developed  and/or  operated  with  some  measure  of  contractor 
support.  For  example,  one  of  the  major  new  initiatives  is  the  C- 
130  ATS  which  is  currently  being  developed  for  the  Military  Airlift 
Command  (MAC)  at  Little  Rock  AFB,  AR,  under  a  contract  with  the 
CAE-Link  Corporation.  In  addition  to  the  C-130  ATS,  each  of  the 
Air  Force  using  commands  is  also  planning,  developing,  and/or 
operating  a  number  of  other  contractor-supported  aircrew  training 
programs  such  as  the  E-3A,  F-15,  and  F-16  (Tactical  Air  Command) ; 
KC- 1 0  (Strategic  Air  Command);  C-5,  C-141,  C-17,  and  Special 
Operations  Forces  (SOF)  ATS  (MAC) . 

Because  of  the  relative  newness  of  the  ATS  concept.  Air  Force 
experience  with  respect  to  the  acquisition  and  management  of 
contractor-supported  total  aircrew  training  systems  is  extremely 
limited.  Most  of  the  Air  Force's  major  ATS  programs  are  still  in 
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the  very  early  stages  of  the  system  life  cycle.  As  a  consequence, 
there  is  relatively  little  empirical  and/or  experiential 
information  to  provide  guidance  for  the  design,  development, 
evaluation,  and  operation  of  new  or  proposed  systems.  In  fact,  in 
most  of  the  systems  reviewed  to  date,  major  command  program 
managers  began  at  ground  zero  and  were  "learning  by  doing,"  except 
for  occasional  consultation  with  personnel  in  other  programs  which 
have  only  slightly  more  experience  than  they  themselves  have. 
Thus,  there  is  a  critical  need  to  systematically  collect,  document, 
and  disseminate  experiential ly  based  "lessons  learned"  to  training 
system  acquisition  offices  and  operational  training  organizations 
to  ensure  a  more  cost-effective  approach  in  the  design, 
development,  and  utilization  of  both  ongoing  and  future  aircrew 
training  systems.  Because  of  the  number  of  aircrew  training 
systems  at  various  stages  of  development  at  the  present  time,  a 
window  of  opportunity  is  available  for  initiating  a  study  of 
lessons  learned  during  various  phases  of  the  training  system  life 
cycle . 

In  response  to  the  need  for  empirical  data,  Armstrong 
Laboratory  established  a  program  to  develop  an  aircrew  training 
system  design/  "lessons  learned"  database.  This  database  is 
intended  for  use  as  an  R&D  resource  and  as  a  source  of  information 
for  the  development  of  user-oriented  guidelines  for  the  cost- 
effective  design,  development,  implementation,  and  utilization  of 
integrated  aircrew  training  systems.  The  objectives  of  this 
program  are; 

(1)  To  collect  "lessons  learned"  by  Air  Force  and  contractor 
personnel  in  selected  Air  Force  ATS  programs  and, 

(2)  To  identify  and  document  key  issucs/problem  areas  which 
might  provide  a  focus  for  the  development  of  a  high  payoff  R&n 
program. 

The  bulk  of  the  data  obtained  to  date  under  this  research 
program  has  been  collected  from  several  major  Air  Force  ATSs  that 
are  serving  as  a  core  group  for  the  identification  of  "lessons 
learned"  and  other  kinds  of  training  system  information.  Some  data 
have  also  been  obtained  from  other  selected  programs  in  both  the 
Air  Force  and  the  other  services.  A  series  of  visits  have  been 
made  to  several  ATS  contractor  and  military  facilities  to  interview 
key  program  personnel.  In  addition,  considerable  time  has  been 
devoted  to  the  review  of  program  documentation  and  other  relevant 
published  information.  As  a  result  of  these  efforts,  a  number  of 
major  ATS  issues/"lessons  learned,"  which  appeared  to  be  common 
across  several  systems,  have  been  identified.  Some  of  the  findings 
to  date  are  documented  and  discussed  in  Rockway  and  Nullmeyer  (in 
press),  with  special  emphasis  on  the  data  obtained  from  the  C-130, 
C-5,  and  KC-10  ATS  programs.  These  particular  programs  are  part  of 
the  core  group  selected  for  continuing  review  and  analysis. 


Purpose  of  This  Paper 


The  purpose  of  this  paper  is  to  capture  some  of  the  more 
salient  experiences  and  "lessons  learned"  by  Lt  Col  Dukes  during 
his  long  involvement  with  the  C-130  ATS  program.  Lt  Col  Dukes  has 
been  personally  and  continuously  involved  with  the  C-130  ATS  from 
the  conceptual  vision  of  the  C-130  MATS  (Model  Ai’'crew  Training 
System)  study  to  the  reality  of  its  operational  implementation. 
The  coauthors  regard  Lt  Col  Dukes  as  a  particularly  valuable 
information  source,  not  only  because  of  his  extensive  experience 
with  the  C-130  ATS,  but  also  because  of  his  detailed  knowledge  of 
other  contractor-supported  training  systems.  Thus,  in  the  opinion 
of  the  coauthors,  he  is  uniquely  qualified  to  provide  a 
particularly  valuable  operational  perspective  v;ith  respect  to  the 
key  issues  and  "lessons  learned"  in  the  C-130  ATS  program.  The 
information  provided  by  Lt  Col  Dukes  is  being  documented  in  this 
form  in  order  to  make  it  readily  available  as  a  source  of  useful 
information  for  personnel  interested  in  the  acquisition, 
development,  implementation,  operation,  or  R&D  on  contractor- 
supported  training  systems. 

This  paper  is  divided  into  five  sections.  Section  I, 
Introduction,  describes  the  purpose  of  this  paper  and  the  larger 
effort  of  which  it  is  a  part.  Section  II,  C-l'JO  ATS  Lessons 
Learned,  provides  a  general  description  of  the  C-130  ATS  program 
and  summarizes  Lt  Col  Dukes'  experiences  and  "lessons  learned."  It 
should  be  noted  that  most  of  the  information  in  Section  II  of  this 
paper  is  an  edited  transcript  of  Lt  Col  Dukes'  remarks  which  were 
recorded  on  audiotape.  Thus,  although  written  in  the  first  person, 
much  of  the  content  is  actually  paraphrased.  During  the  course  of 
the  meeting,  Lt  Col  Dukes  discussed  his  experiences  and  "lessons 
learned"  in  a  number  of  functional  areas  of  the  C-130  ATS  program, 
inv,]uding  courseware,  training  management,  test  and  evaluation, 
quality  assurance,  and  configuration  management.  Ho  concluded  his 
presentation  with  a  number  of  generic  lessons  learned  which  provide 
a  profitable  source  of  guidance  for  all  military  organizations 
involved  in  the  acquisition  of  contractor  developed  and  operated 
training  systems.  Section  III  consists  of  some  orief  coin_li^ding 
remarks  by  the  editors.  Section  IV  contains  a  list  of  references 
from  the  body  of  this  paper.  Section  V  contains  a  larger 
bibliography  of  reports  which  can  provide  useful  information  for 
personnel  interested  in  the  design,  development,  and  evaluation  of 
ATSs. 


II.  C-130  ATS  LESSONS  LEARNED 

(Note:  Except  for  the  material  entitled  "C-130  System 
Description"  under  Subsection  System  Overview,  this  entire  section 
consists  of  an  edited  version  of  Lt  Col  Dukes'  remarks.) 
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System  Overview 


C-130  ATS  System  Description 

The  C-130  ATS  is  an  integrated  contractor-supported  training 
sy'tam  which  is  being  developed  under  a  contract  with  CAE-Link  to 
provide  ground-based  training  for  all  C-130  aircrew  positions  and 
engine  run  maintenance  personnel.  The  C-130  ATS  contract  includes 
28  courses  for  the  Department  of  Defense  (DoD)  formal  school  at 
Little  Rock  AFB,  AR,  and  all  C-130E  and  H  model  continuation 
training.  The  system  includes  the  optimized  use  of  existing 
training  assets,  including  ten  C-130  weapon  system  trainers  (Wr>Ts)  , 
tv;o  cockpit  procedure  trainers  (CRTs)  ,  and  several  part-task 
trainers  (PTTs)  which  were  furnished  to  the  contractor  by  the 
government  as  is.  It  also  includes  all  maintenance  and  logistic 
support  for  the  WSTs  and  other  PTTs  within  the  program.  It 
includes  total  system  management  of  all  ground-based  training  using 
computerized  management  tools,  all  scheduling,  and  all  training 
scenarios  for  the  flying  environment.  It  also  includes  a  training 
continuum  which  begins  with  entry  into  the  formal  school  and  ends 
v.’ith  either  transfer  out  of  the  weapon  system  or  retirement. 

Under  the  C-130  ATS  concept,  the  contractor  is  responsible  for 
the  entire  Instructional  System  Development  (ISD)  process  from 
beginning  to  end,  including  formative,  summative,  and  operational 
evaluations.  The  contractor  is  responsible  for  the  development  and 
production  of  all  courseware,  all  ground  instruction,  all  hardware 
modifications,  and  any  new  software  development.  They  also  are 
responsible  for  the  total  operation,  maintenance,  and  support  of 
the  ground-based  training  system,  all  student  management, 
administration,  configuration  management,  and  quality  assurance. 
The  primary  product  or  output  of  the  C-130  ATS  is  a  "guaranteed 
student . " 


Guaranteed  Stude nt 

One  of  the  most  important  and  least  understood  requirements  of 
contracted  ATSs  is  the  "guaranteed  student."  The  most  prevalent 
assumption  appears  to  be  that  the  contractor's  primary  obligation 
is  to  ensure  that  students  are  able  to  pass  an  Air  Force- 
administered  flight  check  following  contractor-administered  ground 
training.  This  assumption  is  not  entirely  accurate.  We  have  to 
educate  both  contractor  and  government  personnel  that  the  check 
ride  is  too  small  a  sample  of  the  aircrew  knowledge  and  skills 
required  to  be  used  as  the  guarantee  for  an  entire  training  course. 
The  contractor  has  to  be  accountable  for  training  all  of  the  course 
objectives  to  whatever  standard  the  government  has  agreed.  In 
turn,  the  government  needs  a  system  to  ensure  that  students  have 
been  properly  trained  on  all  objectives,  down  to  the  knowledge 
level  in  academics. 
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This  is  startling  news  to  some  contractors  who  have  not 
completely  thought  the  issue  through.  They  do  not  understand  the 
magnitude  of  the  testing  and  performance  tracking  required.  For 
example,  the  C-130  flight  engineer  course  alone  has  484  objectives, 
and  that  is  just  one  of  over  20  courses  in  ^he  C-130  ATS. 
Generally,  I  point  out  the  need  to  provide  some  kind  of  report  to 
the  government  to  document  that  all  of  the  training  objectives  have 
been  met.  I  further  note  that  with  any  volume  at  all  they  will 
need  to  automate  the  process  to  make  it  both  manageable  and  timely. 
This  is  necessary  so  they  can  prove  to  the  government  that  all 
students  have  met  the  criteria  for  all  of  the  training  objectives 
prior  to  being  recommended  for  a  check  ride.  Until  that  is  done, 
the  student  should  not  be  put  up  for  a  check  ride  because  the 
government  does  not  know  what  was  actually  trained  and  what  was 
not.  This  is  essential  because  the  government  is  no  longer 
actively  involved  in  that  part  of  the  training  process  under  the 
contractor's  control. 

I  also  think  that  a  lot  of  people  do  not  understand  that  a 
large  percentage  of  the  critical  things  one  needs  to  know  as  a  crew 
member  are  not  trained  or  tested  in  the  airplane.  For  example, 
such  things  as  electrical  malfunctions,  hydraulic  malfunctions, 
engine  fires,  ditchings,  etc.  You  may  use  the  aircraft  for 
practicing  an  engine-out  approach  or  a  windmill  taxi-start,  but 
that's  about  it  for  malfunctions.  The  other  things  are  usually 
signed  off  to  the  required  standard  in  the  simulator  and  not  on  an 
aircraft  check  ride. 

Gu a ranteed  Training  System 

In  addition  to  guaranteeing  the  capabilities  of  the  student 
graduates,  the  training  "factory"  itself  must  be  guaranteed.  This 
concept  did  not  seem  to  be  a  consideration  in  some  of  the  earlier 
ATSs.  No  one  in  the  military  appeared  to  care  about  the  factory, 
since  they  felt  that  was  the  contractor's  problem.  But  if  the 
military  has  to  recompete  the  system  and  another  contractor  takes 
over,  it  is  crucial  that  both  the  new  contractor  and  the  military 
know  what  they  have.  This  really  hit  home  for  me  when  the  Navy- 
terminated  their  original  contractor  for  the  E-6  training  program 
in  Waco,  Texas.  The  transition  to  a  new  contractor  was  made  more 
difficult  and  more  costly  because  no  one  had  checked  and  validated 
through  quality  assurance,  the  factory.  This  will  be  discussed 
more  when  the  importance  of  quality  assurance  and  configuration 
control  is  discussed. 


Coursewa  re 

Two  major  facets  of  courseware  pi'oduction  in  the  C-130  ATS 
[itogram  will  be  covered.  First,  the  development  process  used  will 
be  (iiscussed,  and  secondly,  our  courseware  production  tracking 
;-,vstem . 
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Courseware  Development  Process 

To  summarize  the  C-130  ATS  courseware  oevelopment  process, 
most  ATS  contracts  have  similar  requirements  although  the 
terminology  used  might  be  different.  For  example,  some  contracts 
call  for  master  task  listings.  In  our  case,  we  have  tasks  and 
objectives  documents.  I  believe  that  the  TTTS  (Tanker  Transport 
Training  System)  has  objectives  and  media  standards  or  something 
like  thrt.  Whatever  it  is,  you  have  to  go  through  a  similar 
process;  that  is,  you  have  to  start  at  the  top  and  go  through  the 
objectives,  select  the  media,  lay  out  your  courses,  and  prepare 
lesson  specifications. 

In  the  MSSR  (Media  Selection  Syllabus  Report),  we  list  the 
units  of  a  course  and  then  for  each  unit  we  break  out  lessons  as 
Lesson  1,  Lesson  2,  Lesson  3  and  so  on.  For  every  lesson  we  list 
the  objectives  as  Objective  1,  Objective  2,  etc.  For  each  lesson 
we  have  no  less  than  two  and,  in  most  cases,  no  more  than  ten 
objectives.  We  define  the  instruction/courseware  for  each  of  the 
objectives  as  a  lesson  segment.  This  system  has  worked  well  for  us 
and  was  a  big  lesson  learned  out  of  C-5,  KC-10,  and  B-1. 

If  I  had  to  do  it  again,  I  would  basically  do  it  exactly  the 
same.  In  some  programs,  a  different  approach  was  used.  For 
instance,  in  one  of  the  programs  that  I  am  familiar  with,  as  soon 
as  they  identified  objectives,  they  started  writing  lessons.  The 
problem  with  this  approach  is  that  each  objective  is  considered 
somewhat  in  isolation.  For  example,  if  an  objective  is  performed 
on  a  Before  Takeoff  Checkl.ist,  do  you  put  "notes,"  "warnings,"  and 
"cautions"  in  there,  or  in  another  lesson?  In  our  case,  we  had  an 
outline  requirement  as  a  contract  deliverable.  So  on  the  first 
page  of  the  lesson  segment,  you  will  see  the  objectives/segments 
and  in  the  outline  it  will  begin  with  Segment  A.  Before  we  start 
writing,  we  agree  that  this  is  the  level  of  detail  to  which  we  are 
going  to  write  the  lesson.  To  save  work  for  the  contractor,  there 
was  a  blanket  statement  on  each  one  of  the  academic  lessons  which 
said,  "If  there  are  any  'notes,'  'warnings,'  and  'cautions,'  they 
will  be  covered  in  that  segment.  Also,  if  appropriate,  you  will 
also  cover  any  switch,  gauge,  or  light  that  is  appropriate  for  that 
system. " 

In  our  initial  cut  at  this,  we  went  down  to  the  individual 
switch  level  on  the  master  task  listing.  We  did  that  based  on  the 
approach  used  for  the  C-130  MATS  (Model  Aircrew  Training  System 
study).  (Note:  For  further  information  on  the  MATS  program,  see 
Fishburne,  Williams,  Chatt,  and  Spears,  1987;  and  Fishburne, 
Spears,  and  Williams,  1987.)  The  MATS  program  identified  100,000 
objectives.  In  the  C-130  ATS,  we  ended  up  with  about  14,000,  and 
even  that  is  a  volume  nightmare  for  the  Training  Management  System 
(TMS) ,  but  100,000  was  totally  unworkable.  After  considerable 
discussion  we  decided  that  we  did  not  want  to  remediate  to  the 
switch  level.  We  felt  that  if  the  student  did  not  know  how  to 
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throw  the  right  switch,  this  was  probably  an  indication  of  a  bigger 
problem.  Therefore,  you  probably  should  be  looking  at  the  whole 
line  on  the  checklist.  Thus,  if  it  was  something  on  the  Before 
Takeoff  Checklist,  you  might  simply  say,  "Set  hydraulic  panel,  set 
fuel  panel,  set  electrical  panel,  and  so  on."  In  some  cases,  there 
might  be  up  to  eleven  steps  on  that  one  step  where  the  student  had 
to  go  across  that  panel  and  set  switches.  So  at  that  point  we 
said,  "OK,  we'll  remediate  him  to  cover  the  entire  step  and  all  the 
switches/procedures  on  that  step." 

However,  after  thinking  about  it  some  more,  we  realized  that 
if  a  student  has  a  problem  with  an  individual  step,  he  probably 
needs  to  review  and  practice  the  whole  checklist.  So  we  finally 
settled  on  the  checklist  level  as  the  starting  point.  A  singular 
checklist  or  a  singular  procedure  became  the  norm  for  hands-on 
objectives.  A  singular  procedure,  whether  it  was  an  emergency 
procedure  or  a  normal  procedure,  equals  one  objective.  We 
remediate  to  the  level  of  a  single  checklist,  procedure  or  single 
air  maneuver.  With  this  approach  you  do  not  list  things  at  the 
level  of  "perform  landings."  If  it  has  an  "s"  on  it,  it  is  too 
high.  It  has  to  be  "perform  100%  flap  landing,  50%  flap  landing, 
no  flap  landing,  cross  wind  landing,  engine  out  landing,  and  night 
landing."  You  spell  out  each  one  and  each  one  has  its  own 
objective.  The  same  applies  to  approaches.  For  example,  you 
should  not  simply  say,  "Perform  non-precision  approaches."  There 
are  all  kinds  of  approaches  and  they  involve  totally  different 
procedures,  so  you  have  to  be  specific.  Taking  the  singular 
approach  really  worked  for  us,  and  we  agreed  upon  the  level  of 
detail  in  the  lesson  specification  document  before  we  started 
writing  the  lesson. 

Another  lesson  learned,  and  something  the  C-17  guys  did  which 
I  think  is  a  waste  of  time  and  money  is  that  they  required  the 
contractor  to  deliver  all  of  the  lesson  specifications  before  they 
Vvrote  the  first  lesson.  Also,  they  wrote  lesson  specs  for  lessons 
that  they  were  not  going  to  start  writing  for  another  year  and 
submitted  them  to  the  government.  The  airplane  has  not  even  been 
to  OT&E  (Operational  Test  and  Evaluation)  yet,  so  many  of  the 
lesson  specs  will  be  OBE'd  (overtaken  by  events)  and  in  my  opinion 
require  major  rewrites.  We  originally  had  a  similar  requirement, 
but  saw  this  as  a  possibility  early  on  and  with  ASD  (Aeronautical 
Systems  Division)  approval,  we  said,  "You  may  not  begin  writing  a 
lesson  until  a  lesson  specification  has  gone  through  an  informal 
government  SME  (Subject  Matter  Expert)  review  and  you  have  an  SME 
signature  on  it.  At  that  point  in  time,  you  can  begin  to  write  the 
lesson,  but  you  won't  have  to  deliver  the  lesson  specification  to 
the  Government  until  thirty  days  before  CRR  (Course  Readiness 
Review)."  This  allows  final  changes  to  the  lessons  to  be  reflected 
in  the  specs  if  required. 

After  a  lesson  is  written,  it  almost  always  requires  some 
modifications.  For  instance,  you  might  research  the  regulations 
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and  ^ind  that  there  is  something  in  MACR  55-130,  C-130  Tactical 
Airlift  Operations,  that  should  be  considered,  so  you  have  to  put  it 
in.  These  are  changes  that  both  the  government  and  contractor 
agreed  needed  to  be  done. 

At  the  last  step  of  the  courseware  review  before  CRR,  we 
conducted  what  we  called  a  "courseware  configuration  audit"  which 
generally  took  about  four  hours  to  complete  and  another  four  hours 
for  the  TMS  crosschecks.  We  took  the  MSSR,  the  lesson  specifi¬ 
cation  and  the  lessons  themselves,  and  we  matched  all  the  objec¬ 
tives  and  the  objective  numbers  along  with  the  test  question 
references,  and  made  sure  that  the  course  guides  and  the  unit  maps 
all  coordinated. 

When  you  follow  a  process  like  this,  it  allows  you  to  start 
with  a  nicely  configured  product.  Then  when  you  make  changes, 
there  is  a  line  in  the  process  which  says,  "Check  the  lesson 
specification  to  see  if  it  is  affected  and  check  the  MSSR." 

The  two  top  level  documents,  the  task  listing  and  the 
objectives  hierarchy,  will  be  done  in  conjunction  with  the  last 
block  of  courses  because  they  are  not  course  specific.  They  cover 
the  whole  system.  Prior  to  the  last  block  CRR,  we  will  take  all  of 
the  MSSRs  that  are  configured  and  check  them  against  the  objectives 
hierarchy  and  the  task  listing  as  part  of  the  last  block's  review, 
and  then  we  will  have  the  whole  system  configured. 

The  formal  school  regulation  (MACR  50-9,  MAC  Formal  Aircrew 
Schools  Management)  requires  us  to  do  that  same  type  of  audit.  So, 
in  my  view,  the  government  in  conjunction  with  the  contractor 
should  continue  to  do  a  similar  annual  audit  to  check  the  match. 
If  you  do  not,  you  end  up  with  a  system  that  will  slowly  grow  non¬ 
concurrent  . 

Our  development  process  involves  55  steps.  These  55  steps  are 
divided  into  three  phases.  The  end  of  Phase  I  is  ITOs  (Individual 
Try  Outs) ,  the  first  T&E  (test  and  evaluation)  test.  The  end  of 
Phase  II  is  an  SGTO  (Small  Group  Try-Outs) ,  and  the  end  of  Phase 
III  is  the  CCA  (Course  Configuration  Audit) ,  which  is  course  work, 
configuration,  and  audit.  In  other  words,  this  is  the  delivered 
baseline.  Of  course,  it  is  not  "finally"  baselined  until  TSRR 
(Training  System  Readiness  Review) ,  but  it  was  configured  as  an 
initial  baseline  at  the  end  of  the  third  Phase.  My  biggest  lesson 
learned  here  is  don't  do  ITOs.  Instead,  do  UTOs  (r  .it-level  Try- 
Outs)  .  In  my  view,  our  ITOs  gave  us  only  two  pieces  of  valid 
information — you  got  the  reading  level  of  the  lesson  and  a  ballpark 
time  hack.  But  the  reason  I  say  do  the  test  at  the  unit  level  is 
that  we  automatically  waived  all  the  unit  tests  and  the  unit 
reviews  and  the  end  of  course  exam  because  you  know  each  student  is 
just  looking  at  one  lesson  at  a  time  during  ITOs.  If  you  gave 
students  a  unit  review,  they  would  not  know  what  you  were  talking 
about,  since  they  hadn't  seen  the  whole  unit.  So  I  recommend  doing 
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try-outs  at  the  unit  level.  In  other  words,  get  a  new  student  that 
has  not  seen  hydraulics  yet  and  give  him  the  whole  hydraulic  unit, 
and  then  give  him  the  unit  quiz  and  the  unit  test  as  a  part  of  the 
UTO. 


The  biggest  problem  we  had  with  ITOs  and  SGTOs  was  scheduling. 
We  started  out  with  2,000  lessons,  and  ended  up  with  about  1,867. 
Our  planned  schedules  were  all  running  over,  especially  CBT 
(computer-based  training) .  However,  if  you  run  them  at  the  unit 
level,  you  would  get  a  better  overall  picture  of  time.  The  biggest 
write-up  we  had  in  SGTOs,  and  you  probably  could  have  predicted  it, 
was  that  our  unit  reviews  were  less  than  desired.  The  contractor 
had  gone  back  to  the  individual  lessons  and  just  reproduced 
material.  A  better  idea  is  to  conduct  the  review  in  a  more 
abstract  form  and  play  with  it  a  little  bit,  like  you  do  in  a  good 
lecture.  ISO's  position  was  that  you  had  to  follow  the  format 
guide.  Our  major  rewrites  occurred  in  the  unit  reviews.  That  is 
a  good  lesson  learned,  I  think.  It  makes  it  easier  on  both  the 
contractor  and  the  government  to  schedule  the  units  as  blocks  and 
requires  fewer  students. 

Quality  Control.  The  point  that  I  wish  to  make  here  is  that 
the  courseware  quality  control  function  (QC)  is  not  the  same  as 
quality  assurance  (QA) .  QC  is  internal.  In  other  words,  that  is 
someone  looking  to  say,  "OK,  this  is  our  style  guide,  these  are  our 
conventions  for  color,"  etc.,  and  that  is  all  QC  is  checking.  They 
are  doing  a  QC  of  the  lesson  in  terms  of  format,  spelling  errors, 
does  it  read  right  and  that  sort  of  thing.  In  no  way,  shape  or 
form  is  that  quality  assurance,  because  QA  has  different  functional 
requirements.  But  I  want  to  make  that  distinction  here  because  you 
have  QC  in  many  different  things.  You  can  have  QC  in  maintenance, 
like  in  your  CLS  (Contractor  Logistic  Support) .  You  can  have  QC  in 
your  TMS  (Training  Management  System) .  Guys  checking  what  a  guy 
programmed  and  that  sort  of  thing,  but  that  is  quality  control. 
Quality  assurance  and  quality  control  on  C-130  ATS  are  apples  and 
oranges.  One  is  internal,  and  the  other  is  external.  That  is  the 
key.  QC  is  internal.  When  I  get  to  the  QA  part  of  the  discussion, 
I  will  spell  out  the  differences  more  explicitly. 

Government  Review.  There  is  a  government  review  in  each 
phase.  In  Step  8,  Step  18  and  Step  42  of  our  55-step  process,  we 
had  government  reviews.  One  of  the  things  that  came  up  early — and 
we  were  saved  by  our  CRR  checklist — is,  "What  if  you  get  to  the 
last  government  review  which  should  mean  all  the  write-ups  should 
be  cleared  and  they're  not  cleared?"  Early  in  the  program,  we 
would  see  as  many  as  50  open  write-ups  when  the  contractor  was 
letting  things  ride.  What  we  did  after  the  second  government 
review  was  to  transfer  them  to  a  CRR  open-action  item  checklist  and 
state,  "If  you  don't  fix  these  by  CRR,  you  don't  pass."  That  is 
why  the  CRR  checklist  that  we  signed  up  to  one  year  before  the 
first  CRR  was  so  important.  It  meant  that  there  was  no  argument. 
For  example,  our  last  CRR  was  30  seconds.  The  one  before  that--we 
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did  the  long  loadmaster  and  navigator  course — was  six  minutes.  If 
there  is  an  open  action  item,  everybody  knows  what  it  is  before  you 
go  in  the  door.  You  just  sit  down  and  run  through  the  script,  sign 
it  and  leave.  "It's  really  important  to  have  that  checklist."  I 
have  given  copies  of  it  to  almost  everyb  ly.  It  will  have  to  be 
tailored  for  each  contractor  to  sign  up  to,  but  it  is  very 
comprehensive.  An  additional  thought  here  is  that  initially  in 
your  first  blocks,  it  is  very  tedious  to  do  all  the  facilities  and 
instructor  CRR  checklist  items,  but  these  will  fall  out  in 
subsequent  blocks  because  you  have  already  checked  them. 

Courseware  Production  Tracking 

Production  tracking,  or  the  lack  of  it,  is  what  caused  one  of 
the  biggest  problems  on  this  program.  The  contractor  did  not  know 
where  they  were  in  terms  of  the  courseware  development  schedule. 
We  also  did  not  know  where  they  were,  because  we  expected  them  to 
use  a  tracking  system  which  we  would  manage  against.  However, 
since  they  did  not  have  one,  they  did  not  know  whether  they  were  a 
month  ahead  or  a  month  behind.  At  first,  they  were  going  to  build 
their  tracking  system  in  the  TMS,  but  that  did  not  work  out.  Then 
they  put  wall  boards  up  and  down  the  halls.  That  lasted  one  week. 
And  they  pulled  them  down.  They  finally  ended  up  with  Timeline  3 
which  did  a  very  good  job.  Through  trial  and  error  we  established 
due  dates  and  timelines  on  about  how  long  it  would  take  to  do  one 
of  those  55  steps.  Of  course,  the  long  parts  are  the  authoring  and 
the  artwork.  But  some  things,  like  simulator  scenarios,  might  take 
three  weeks  to  develop  and  a  half  day  to  test.  We  got  our 
estimates  massaged  enough  so  that  now  it  works  fine  and  there  are 
no  problems. 

To  stay  on  track,  the  production  information  was  all  loaded  in 
Timeline.  They  have  three  people  full-time  that  work  Timeline. 
They  take  a  sheet  of  paper  with  all  the  lesson  production  steps  on 
it  and  draw  a  black  line  out  to  the  step  that  it  was  at.  The 
personnel  on  the  night  shift  would  enter  all  of  it  into  Timeline 
and  update  the  schedule.  Because  we  have  had  as  many  as  1000 
lessons  in  production  at  one  time,  it  was  critical  to  get  an 
automated  system  running. 

This  did  not  happen  until  about  nine  months  into  the  program, 
and  I  would  get  a  copy  which  said,  "Here's  where  we're  supposed  to 
be  and  here's  where  we  are."  Initially,  when  there  were  still 
problems,  I  built  a  manual  system  to  track  their  progress,  so  when 
things  came  over  for  government  review,  and  they  would  say  you  are 
supposed  to  get  20,  and  I  would  get  2,  I  would  say  you  are  18 
short.  The  bow  wave  kept  getting  bigger.  And  every  week  I  would 
raise  the  issue,  and  they  would  change  the  schedule.  Anyway,  what 
finally  happened  was  that  they  implemented  Timeline  and  put  all  the 
due  dates  in  and  tracked  against  those  dates,  and  it  worked  fine. 
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with  Timeline,  we  set  up  a  three-day  tracking  window.  With  a 
three-day  window,  if  the  lesson  got  ahead  by  three  days,  you  could 
actually  slow  it  down  and  play  catch-up  with  other  lessons  and  do 
some  tradeoffs,  but  if  it  got  behind  by  three  days,  they  put  a  red 
tag  on  it.  When  we  have  a  stack  of  lessons  to  review,  the  first 
thing  my  men  would  do  is  look  at  the  due  date  and  prioritize  it. 
If  it  has  a  red  tag  on  it,  it  goes  on  the  top  of  the  heap,  and  that 
has  worked  very  well. 

We  have  told  them  the  schedule  is  a  stake  in  the  ground. 
They  cannot  simply  change  the  due  dates.  Initially,  when  we  were 
just  beginning,  they  had  three  days  to  author  a  simulator  scenario, 
and  the  men  over  there  were  saying,  "No  way."  Because  it  actually 
took  weeks  to  write  them,  including  such  things  as  the  malfunction 
codes,  the  scenarios,  the  weather,  and  the  integration  between  crew 
members,  it  is  not  just  a  simple  thing  to  put  one  together.  My 
"lesson  learned"  in  courseware  tracking  is  that  one  should  gather 
data  to  find  out  how  long  it  takes  to  complete  a  job.  Then  use  it 
as  a  basis  for  assessing  whether  or  not  what  the  contractor  tells 
you  is  in  the  ballpark. 

There  is  an  upper  limit  with  respect  to  the  number  of  lessons 
in  production  that  can  be  tracked  manually.  In  my  opinion,  the 
break  point  is  about  400  lessons,  although  this  may  change  as  a 
function  of  lesson  complexity.  Above  that  number,  you  need  an 
automated  system.  When  they  started.  Block  1  had  44  lessons  and 
Block  2  add  another  100.  When  we  reached  the  instructor  school  in 
Block  3,  the  schedule  hit  the  fan.  We  just  lost  it,  and  Block  4, 
the  largest  block,  was  unreal.  We  have  had  over  1000  lessons  in 
production  at  one  time,  all  somewhere  between  Step  1  and  Step  55. 

Many  people  do  not  understand  that  building  lessons  for  a 
course  is  not  like  building  simulators.  If  you  are  building  1,000 
simulators,  every  little  red  wire  that  goes  into  this  box  is 
exactly  the  same  for  all  simulators.  On  the  other  hand,  there  are 
no  two  sentences  in  any  course  that  are  the  same;  and  no  two  people 
that  review  it  will  look  at  it  the  same.  One  of  the  biggest 
"lessons  learned"  was  the  need  for  a  step  to  run  a  lesson  by  the 
instructors.  We  did  this  because,  after  they  ran  the  first  couple 
of  courses,  the  instructors  criticized  the  courseware  saying,  "We 
don't  teach  this  way.  It  doesn't  flow  for  me  in  the  classroom." 
Of  course,  on  the  Air  Force  side,  we  were  saying,  "I  told  you  so," 
because  the  contract  would  not  let  us  comment  on  flow,  only  on 
content.  However,  when  hundreds  of  ICPs  (Instructional  Change 
Proposals)  were  submitted  on  the  lessons,  they  found  out  that  it 
was  costing  them  money.  If  they  had  done  that  instructor  review  in 
the  first  place,  they  would  not  have  had  to  redo  quite  so  much. 
So,  getting  a  process  and  production  tracking  system  in  place  that 
has  been  validated  is  really  important  if  you  are  going  to  make  a 
schedule,  and  make  it  within  budget. 


11 


I  think  that  our  estimates  of  courseware  development  times 
have  some  generality  for  other  systems.  The  only  ones  that  might 
change  significantly  would  be  CBT.  This  is  particularly  the  case 
if  you  add  in  IVD  (Interactive  Video  Disk) ,  which  we  did  not  have! 
If  you  have  IVD  and  do  a  large  amount  of  branching,  the  development 
time  will  go  exponentially  out  of  sight.  When  we  started  out,  it 
took  about  468  hours  to  develop  a  CBT  lesson.  However,  by  the  time 
the  graphics  library  was  built  and  the  packaging  teams  were 
humming,  where  they  knew  the  key  strokes  and  how  to  really  move,  we 
were  down  to  about  320  or  328  hours  per  lesson.  It  seems  that 
these  averages  hold  up  no  matter  how  complex  the  lesson.  So  that 
was  a  good  "lesson  learned." 

Training  Management  System 

The  C-130  ATS  TMS  consists  of  eight  modules: 

1.  Administrative  management 

2 .  Resource  management 

3.  Curriculum  management 

4.  Scheduling  management 

5.  Performance  measurement 

6.  Reports 

7.  Configuration  management 

8.  Logistics  management 

The  biggest  lesson  learned  in  the  TMS  was  that  we  and  the 
contractor  underscoped  the  required  system  capacity,  because  we 
underestimated  the  amount  of  data  involved  in  building  an 
integrated  system.  When  we  started  out,  we  were  going  to  do  the 
whole  thing  on  two  hard  drives,  with  800  megabytes  each  running  on 
two  AT&T  3B2600S.  We  now  use  five  at  the  formal  school.  The 
curriculum  module  itself  now  requires  two  to  handle  all  the 
courseware  and  CBT  information. 

As  noted  previously,  we  have  one  integrated  TMS  database  that 
handles  the  bulk  of  the  instructional  system  functions.  In  my 
view,  some  of  the  other  ATSs  that  did  not  follow  this  approach  are 
in  for  some  tough  times  in  the  future.  For  example,  in  the  C-17 
they  are  writing  the  courseware  on  an  AIS  II,  and  their  TMS  is  part 
of  the  CMI  (Computer-Managed  Instruction)  on  that  system.  They 
will  probably  not  have  the  capacity,  speed  and  flexibility  that  we 
have  with  a  single  integrated  relational  database.  The  big 
advantage  of  the  way  we  have  done  it  is  that  all  of  the  modules  can 
talk  to  each  other,  since  everything  is  in  one  TMS  relational 
database.  I'll  cover  each  of  the  TMS  modules  in  turn. 

Administrative  Management  Module 

The  administrative  management  module  does  three  basic  things; 
(1)  It  runs  the  registrar  function,  (2)  it  handles  all  of  the 
security  requirements,  and  (3)  it  transfers  data  between  the  sites. 
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For  example,  when  a  student  graduates,  the  administrative 
management  module  sends  the  students'  files  over  a  modem  to  the 
gaining  unit.  So,  it  keeps  track  of  the  crew  force  in  terms  of 
their  records.  We  will  also  handle  the  historical  documents 
through  this  module,  although  that  has  not  yet  been  implemented. 
We  are  handling  the  essential  files  first.  We  also  have  to  make 
some  decisions  about  what  we  will  keep  on-line  and  what  will  be 
downloaded  and  archived  on  tape. 

Resource  Management  Module 

The  resource  management  module  is  where  you  store  information 
about  the  system  resources.  For  instance,  it  might  store  Marty 
Rockway  as  an  instructor.  It  also  stores  what  he  has  been  trained 
to  teach;  e.g.,  perhaps  he  can  teach  in  the  classroom,  but  he  has 
not  been  checked  out  on  the  simulator.  On  the  other  hand.  Bob 
Nullmeyer  can  go  to  the  flight  line  and  teach  anything  that  he  is 
certified  to  instruct.  If  you  try  to  log  him  into  a  class  as  an 
instructor  for  a  particular  lesson  and  he  has  not  been  certified 
for  that  lesson,  the  module  will  not  let  you  schedule  him.  It 
keeps  track  of  all  of  the  instructors  and  what  they  arc  qualified 
to  teach.  It  also  keeps  track  of  all  facilities,  such  as 
classrooms  and  briefing  rooms,  so  that  if  one  is  being  used  it 
cannot  be  double-booked.  It  does  all  of  the  checks  and  balances 
there . 

Early  on,  we  had  a  tendency  to  forget  that  the  simulator 
briefing  rooms  were  used  for  both  prebriefs  and  postbriefs. 
Because  we  run  extensive  prebriefs  and  postbriefs,  overlap  problems 
may  occur  if  timing  is  not  watched  closely.  However,  with  proper 
scheduling,  one  crew  should  be  prebriefing  while  another  crew  is 
still  in  the  simulator.  Other  things  that  the  module  tracks  are 
all  of  the  hardware,  including  all  aircrew  training  devices.  The 
module  also  stores  the  CBT  terminals  and  knows  how  many  people  can 
get  in  there  and  at  what  time.  It  also  keeps  track  of  the  learning 
center. 

The  other  thing  that  the  module  does — and  this  is  a  "lesson 
learned" — is  store  the  configuration  of  all  training  devices.  This 
would  be  important,  for  example,  if  we  did  a  SONS  (Self-Contained 
Navigation  System)  modification  to  one  of  our  simulators.  Then,  if 
a  student  went  through  a  nonSCNS  courseware,  you  could  not  put  him 
in  a  SCNS  box.  Thus,  it  would  not  let  you  schedule  against  a  box 
that  was  in  a  configuration  that  did  not  meet  the  curriculum. 

Curriculum  Management  Module 

The  curriculum  management  module  is  one  of  the  biggest  modules 
we  have.  It  contains  all  of  the  master  task  listings,  objective 
hierarchies,  MSSRs,  lesson  specifications,  lesson  materials,  etc. 
It  also  has  CBT  information.  We  write  up  through  the  s._oiy-boara 
phase  on  the  TMS .  For  configuration  management  you  need  only  to 
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configure  off  your  storybook  documentation,  because  that  is  where 
you  have  logged  the  frame  numbers,  text  segments  and  so  forth. 
When  configuration  management  interfaces  is  discussed  later,  you 
will  see  how  this  all  ties  together.  It  is  really  slick.  With 
over  1,800  lessons  to  manage,  that  is  why  there  is  so  much  in  the 
curriculum. 

In  curriculum  management,  when  we  do  the  MSSR  unit  listing, 
the  complete  flow  of  the  course  is  given,  and  that  is  where  we 
actually  initiate  the  student’s  schedule.  The  scheduling  model 
interfaces  and  pulls  it  off,  but  you  may  have  to  add  in  a  few 
things  each  time,  however,  like  lunch.  I  think  I  told  you  before 
that  we  forgot  to  include  lunch  on  the  first  two  days  that  we  ran 
SGTOs.  It  didn't  take  long  for  the  student  to  let  us  know  we  had 
a  problem. 

Also,  another  big  "lesson  learned"  is  that  the  computer  does 
not  know  whether  you  are  at  12:00  a.m.  or  p.m.  until  12:01.  This 
is  because  it  runs  a  24-hour  clock  against  each  day;  12:01  is  a 
whole  new  day.  If  a  student  is  going  to  the  simulator  and  he  gets 
nut  at  midnight,  the  computer  might  schedule  him  for  a  CBT  lesson 
an  hour  later.  We  had  to  artificially  install  crew  rest  require¬ 
ments  and  bed  rest  so  a  student  could  not  be  scheduled  until  8:00 
or  10:00  the  next  morning.  The  computer  knows  that  you  are  done 
with  Tuesday  and  now  you  are  up  for  Wednesday,  so  you  have  to  put 
that  in. 

Now,  let  me  explain  to  you  how  the  system  works  day-to-day. 
Marty  Rockway  logs  into  the  school.  When  he  logs  into  the  school, 
he  will  fill  out  a  card,  and  it  says,  "I'm  here  to  take  the  CIQ 
(Copilot  Initial  Qualification)  course.  Class  8806."  The  system 
automatically  checks  that  his  flight  physical  is  current  and  he  has 
been  through  the  physiological  training  and  has  all  the 
prerequisites  to  cover  the  course.  If  not,  it  raises  a  flag  and  we 
x-ry  to  work  that  out.  After  he  is  inprocessed,  the  night  shift 
enters  all  of  the  data  into  the  computer.  The  next  day  when  he 
returns,  he  receives  his  class  schedule.  Then,  the  administrative 
management  module  goes  down  to  the  curriculum  management  module  and 
retrieves  all  of  the  objectives  in  the  course  for  which  Marty 
signed  up.  It  lists  them  in  the  order  of  the  MSSR,  starting  with 
academics,  and  inserts  every  single  objective  against  his  name. 
Then,  it  is  the  performance  measurement  module's  job  to  log  them 
off  and  keep  track  of  when  Marty's  ready  for  his  check  ride. 

Scheduling  Management  Module 

The  next  interface  is  the  scheduling  management  module  (SMM) . 
The  scheduling  management  module  retrieves  only  the  information  it 
needs  from  the  other  modules,  such  as  the  curriculum  management 
module  and  the  resource  management  module,  in  order  to  accomplish 
its  task.  After  you  have  been  logged  into  the  system,  the  SMM  takes 
over.  It  is  run  by  '■he  schedulers  (there  are  three  full-time 


schedulers)  and  they  begin  with  the  baseline  curriculum.  The  whole 
curriculum  is  on  the  screen,  and  there  is  a  little  asterisk/cursor 
you  move  down  and  then  key  in  all  of  the  lessons  you  want  to 
schedule.  We  then  print  out  a  two-week  schedule,  because  one  any 
longer  than  that  would  have  to  be  changed  because  of  remediation. 
You  get  a  two-week  schedule  on  day  one  and  will  know  what  you  will 
be  doing  every  hour  for  the  next  two  weeks.  It  will  list  your 
lesson,  classroom  number,  etc.  It  will  also  give  you  the 
instructor's  name.  When  required,  you  will  receive  a  new,  revised 
schedule.  If  you  enter  remediation,  the  SMM  will  update  your 
schedule  and  project  it  out.  If  you  have  real  problems,  washback, 
sickness,  etc. ,  your  schedule  will  be  adjusted  to  produce  some 
fillers.  Right  now  the  scheduling  management  module  will  produce 
an  individual  student  schedule  as  well  as  a  schedule  for 
instructors.  This  means  that  if  I  am  an  instructor,  it  will  give 
me  exactly  what  I  will  be  doing  for  the  next  two  weeks,  where  and 
when  to  report,  vhat  lesson  I  teach,  so  I  can  plan  for  it. 

We  are  still  managing  the  basic  formal  school  by  blocks  or 
class  of  students.  There  is  some  capability  for  individualized 
management  and  some  capability  to  test-out  of  parts  of  a  course. 
Because  of  the  volume  of  students  and  facilities  limitations,  we 
are  relegated  to  managing  students  in  class  blocks.  However,  there 
are  some  self-paced  areas  like  CBT  where  if  you  complete  a  lesson, 
you  can  get  ahead,  leave,  or  come  back  at  night  to  do  some  more. 
As  you  know,  different  people  work  at  different  rates  and  CBT 
allows  for  that.  In  fact,  the  CBT  facility  is  open  until  10  p.m. 
for  this  reason.  However,  CBT  is  the  only  area  in  which  you  can 
leave  and  go  to  the  BX,  then  come  back  that  night  and  do  the 
lesson.  The  only  requirement  is  that  you  finish  it  before  the  next 
lesson  for  which  it  is  a  prerequisite.  This  is  because  the  system 
will  not  let  you  log  on  to  a  lesson  for  which  you  have  not  met  the 
prerequisites. 

The  SMM  still  does  not  print  out  a  standard  daily  flight 
schedule.  Such  a  schedule  would  show,  for  example,  that  Dukes  and 
Bob  and  Marty  are  going  to  Jackson,  Mississippi,  they  have  an  8:00 
takeoff,  the  fuel  is  36,000,  the  load  is  X  pounds,  and  so  forth. 
The  data  to  do  this  are  all  in  the  system,  but  the  TMS  does  not 
currently  have  a  program  to  bring  it  together  and  print  it  out 
line-by-line  on  a  daily  schedule  basis.  They  are  working  on  this 
now,  and  the  daily  flying  schedule  is  due  out  later  this  year.  We 
have  been  running  a  test  at  Dyess  AFB  on  our  squadron  scheduling 
which  to  date  has  been  going  well. 

There  is  a  possible  interruption  brewing  in  this  area  at  MAC 
Headquarters.  The  DOO  (Operations  Center)  staff  has  briefed  the 
new  CASS  (Computer  Assisted  Squadron  Scheduling)  software  to  the  DO 
(Deputy  Commander  for  Operations)  and  it  is  supposed  to  hook  up  to 
the  numbered  Air  Force  and  MAC  scheduling  that  goes  all  the  way  up 
to  the  Air  Staff.  The  DO  point  of  contact  said  in  both  cases  the 
interfaces  work,  and  he  thinks  that  CASS  is  slightly  ahead  of  where 
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we  are  because  we  just  finished  CDR  (Critical  Desian  Review)  last 
week  and  are  not  yet  into  the  detail  of  working  AFJRMS  (Air  Force 
Operations  Resource  Management  system).  I  think  we  have  defined 
the  requirement;  we  just  have  not  accomplished  any  programming. 
The  CASS  idea  is  valid  if  you  are  trying  to  standardize  the 
command;  in  other  words,  the  C-5  guys,  the  141  guys  and  everybody 
that  interfaced  with  MAC  will  hook  up  because  it  is  written  in  a 
standard  language.  The  latest  word  that  I  am  getting  is  if  we  go 
to  CASS,  we  would  terminate  C-130  ATS  squadron  scheduling. 

Performance  Measurement  Module 


The  performance  measurement  module  (PMM)  in  my  view  is  the 
most  critical  module  in  the  ATS  and  one  that  has  been  designed  and 
built  from  the  ground  up.  It  is  the  module  that  checks  to  make 
sure  the  student  is  signed  off  against  all  of  the  objertives'-f rom 
academics,  through  all  of  the  hands-on  training  right  up  through 
the  check  ride.  We  have  a  bubble  sheet  that  gives  us  feedback  that 
is  tied  to  the  objectives.  The  volume  is  at  a  point  now  that  it  is 
starting  to  give  us  feedback  to  do  overall  curriculum  assessments 
and  refinements. 

Currently,  the  PMM  is  handling  mainly  student  data,  but  we  are 
starting  to  get  some  system  level  performance  data  out  of  it  as 
well.  We  have  created  some  system  level  performance  trend  graphs 
as  part  of  summative  evaluation.  One  covers  check  ride  results, 
Qls,  Q2s,  and  Q3s.  We  are  also  trending  late  graduates  and  student 
deficiency  reports,  i.e.,  things  above  the  level  of  the  individual. 
We  are  still  growing  in  this  area.  Next  month,  all  of  these  T&E 
automated  reports  are  due. 

The  student  training  report  is  generated  out  ot  the 
performance  management  module.  When  a  student  first  logs  in,  if 
you  pull  his  report  it  will  show  all  of  the  objectives  in  the 
course  against  his  name.  As  the  student  progresses  through  the 
course,  the  entry  for  "objectives  not  trained,"  will  start 
reducing.  There  is  another  section  that  lists  all  of  his  academic 
scores  on  his  unit  tests  and  end  of  course  exam.  If  a  student  does 
not  meet  criteria  on  a  hands-on  objective  and  requires  more 
training,  that  will  be  on  the  record  also,  and  it  will  show  where 
he  is  weak.  If  the  student  does  meet  criteria  on  time  or  falls 
below  criteria  at  a  later  date,  that  also  will  be  on  the  record. 
Additionally,  any  objective  that  is  remediated  becomes  a  part  of 
the  record. 

We  pull  the  student  training  report  when  the  Air  Force  flight 
instructor  comes  down  from  the  flight  line  to  observe  a  student  in 
the  simulator  before  he  goes  to  the  aircraft.  This  is  particularly 
critical  for  new  pilots,  since  the  Air  Force  instructor  no  longer 
trains  his  students  in  both  the  simulator  and  the  aircraft.  Thus, 
the  simulator  observation  provides  the  instructor  with  a  preview  of 
the  student's  capabilities  prior  to  the  first  aircraft  training 
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sortie,  and  the  report  provides  this  overall  view  of  his  simulator 
performance.  In  fact,  we  will  probably  end  up  with  initial 
qualification  course  pilots  being  observed  twice  by  their  Air  Force 
flight  instructors  in  the  simulator  before  they  go  to  the  aircraft. 
The  other  crew  positions  and  other  pilot  courses  win  probably  be 
observed  only  once  or,  in  some  cases,  not  be  observed  prior  to 
their  first  aircraft  sortie  because  there  are  fewer  safety 
concerns . 

Since  new  initial  qualification  pilots  have  the  stick  in  the 
aircraft,  it  is  important  that  the  instructor  has  the  opportunity 
to  see  whether  he  is  really  comfortable  with  the  student's  skill  in 
the  simulator  before  they  go  out  to  fly.  For  some  of  the  other 
positions,  e.g.,  loadmaster,  such  observation  prior  to  the  first 
flight  is  not  as  crucial  since  the  instructor  will  be  standing 
right  alongside  the  student  in  the  aircraft.  In  any  case,  we  pull 
the  report  when  the  flight  instructor  meets  the  student,  so  the 
instructor  can  check  on  how  the  student  has  performed  in  academics 
and  hands-on. 

The  next  time  the  student  training  report  is  pulled  is  just 
before  the  student  is  recommended  for  the  check  ride. 
Incidentally,  this  is  mandatory,  so  we  have  written  some  QA 
procedures  to  check  it.  Thus,  if  I  were  a  student,  and  it  is  time 
for  my  last  aircraft  flight  before  the  check  ride,  the  flight 
instructor  would  pull  my  report.  The  only  things  that  should  be 
open  on  the  report  at  this  point  are  those  events  that  the 
instructor  plans  to  cover  and  sign  off  that  day.  If  there  is 
anything  else  open,  such  as  some  landing  condition  that  needs  to  be 
tested  in  an  aircrew  training  device  (ATD) ,  we  inform  the 
contractor.  We  also  check  the  grade  folder  to  see  if  it  is  just  an 
entry  error  or  if  the  student  really  needs  to  go  back  for  training. 
If  training  is  required,  the  contractor  must  provide  it  before  a 
check  ride  is  authorized.  We  now  have  a  procedure  where  the 
contractor's  instructor  operations  office  pulls  those  checks, 
comparing  the  data  in  the  grade  folders  with  that  in  the  TMS  for 
discrepancies.  If  the  student  has  a  clean  sheet,  i.e.,  no 
objectives  untrained,  no  objectives  not  tested,  then  in  my  opinion 
that  is  when  the  student  guarantee  is  fulfilled! 

Let  me  discuss  one  of  the  things  we  have  already  pulled  out  of 
our  T&E  data.  We  went  back  one  year  and  determined  the  average 
percentage  of  check  ride  failures  for  the  last  year.  If  the  C-130 
ATS  meets  or  does  better,  then  we  know  that  the  course  has  met  the 
TSRR  criteria.  It  turns  out  that  in  the  basic  navigator  course, 
30%  of  the  students  received  Q2s.  Looking  deeper,  we  found  that 
was  not  as  bad  as  it  looked.  The  reason  for  the  deficient  rating 
was  usually  a  pacing  timing  error  with  the  SNS  (Satellite 
Navigation  Station).  In  thirty  minutes  they  are  signed  off  and  out 
the  gate,  and  now  it  is  the  contractor's  responsibility  to  sign 
that  off.  These  are  some  of  the  goals  we  are  trying  to  achieve  in 
this  new  management  system. 
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Many  people  do  not  fully  understand  the  importance  of  the  PMM. 
This  goes  back  to  our  previous  discussion  about  the  meaning  of  the 
student  guarantee.  This  is  where  you  must  verify  the  guarantee. 
You  always  have  a  backup,  of  course.  You  can  always  pull  the 
student  grade  folders  and  get  the  information  that  way.  But  this 
is  not  feasible  when  you  have  400  students  on  site,  on  any  given 
day,  and  the  government  does  not  have  visibility  into  three-fourths 
of  the  course.  This  seems  to  suggest  that  the  government  needs 
something  that  is  much  more  efficient  than  a  labor-intensive  manual 
system  in  order  to  know  what  is  going  on.  The  PMM  gives  you  an  on¬ 
line  tool  for  tracking  student  performance  and  generating  relevant 
reports  in  pT-actically  real  time. 

Additionally,  we  have  regular  QA  checks  on  the  system  to 
eliminate  pencil  whipping.  We  pull  the  training  folder  and  compare 
it  to  the  computer-generated  report  to  see  if  they  match.  If  there 
is  a  mismatch,  we  write  them  up.  We  have  caught  a  few 
discrepancies.  In  fact,  we  have  a  process  now  where  each  and  every 
grade  folder  is  checked  after  graduation. 

Rt'^ports  Module 

There  are  18  reports  on  the  CRR  checklist  that  are  duo  at 
different  times.  All  but  two  of  them  are  formal  school  reports 
that  we  have  always  generated.  The  data  for  these  reports  now  come 
out  of  the  TMS  or  out  of  the  contractor's  registrar  function.  The 
most  significant  report,  and  the  one  on  which  we  have  worked  the 
hardest  through  PDRs  (Preliminary  Design  Reviews)  and  CDRs 
(Critical  Design  Reviews),  is  the  student  training  report. 

Before  CRR,  we  accomplish  our  final  configuration  of 
courseware  and  sign  off  all  of  the  lessons.  During  this  process, 
we  match  everything.  We  take  the  MSSR,  unit  tests,  and  end-of- 
course  exams  and  compare  them  with  the  student  training  report  and 
run  it  two  or  three  different  ways.  First  of  all,  we  run  it  with 
no  objectives  accomplished  and  make  certain  that  every  single 
objective  on  the  student  report  matches  what  is  in  the  actual 
course.  They  are  required  to  match  right  down  the  line.  The  next 
thing  we  do  is  take  the  unit  tests  and  check  the  numbers  to  make 
sure  they  are  identical  to  their  respective  objectives.  The  next 
check  that  we  pull,  which  is  very  important,  is  to  look  at  the 
tests,  the  bubble  sheets,  and  the  student  training  report  to  ensure 
they  match. 

The  process  that  takes  place  in  the  system  is  as  follov;s.  The 
student  takes  the  test  and  fills  out  a  bubble  sheet.  The  bubble 
sheet  goes  into  an  optical  scanner  whci»;  it  is  read  and  fed  into 
the  PMM  and  graded.  From  the  PMM,  software  then  generates  the 
training  report.  In  our  configuration  audit,  we  check  the  whole 
thing  from  A-Z.  Basically,  we  are  checking  to  see  what  is  marked 
on  the  bubble  sheet  and  if  the  bubble  sheet  is,  in  fact,  the  right 
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answer,  and  what  we've  already  configured  at  our  courseware 
configuration  audit  is  absolutely  correct. 

We  have  found  a  few  problems  on  how  the  machine  graded  the 
student.  I  mean  the  student  training  report  said  he  got  it  right 
when  he  really  got  it  wrong  and  it  is  a  data  entry  problem.  Most 
of  this  is  because  the  last  thing  we  change  is  the  courseware,  and 
then  you  have  to  change  the  appropriate  test  question,  and  they  did 
not  get  the  same  data  changed  in  the  TMS.  This  is  generally 
because  we  pull  this  check  at  the  last,  and  we  pull  it  off  the  live 
database . 

One  of  the  important  things  you  will  want  to  make  a  note  of  is 
the  question,  "Who  is  responsible  for  all  of  the  data  the  TMS 
gets?"  Is  it  the  software  guys  since  they  are  writing  all  of  the 
software  and  entering  it,  or  is  it  ISD  because  all  of  the 
configured  documents  equal  the  data?  Now,  how  do  you  get  the  data 
into  the  TMS?  Some  of  it  is  transferred  straight  across,  but  a  lot 
of  it,  like  test  questions,  etc.,  are  actually  keyed  in  because  it 
has  to  match  the  software  hooks  at  the  right  time  and  place.  Also, 
who  is  responsible  for  the  correctness  of  the  data?  ISO's  position 
was,  "When  I  get  to  configuration,  I'm  done."  TMS  said,  "Not  me, 
I'm  just  software.  I  just  make  sure  the  software  runs  properly." 
Anyway,  it  ended  up  landing  in  ISD  because  it  has  to  pass  the  hard 
copy  checks.  They  have  two  people  full  time,  to  keep  all  of  the 
TMS  data  current.  They  are  part  of  the  courseware  configuration 
team . 

When  I  try  to  explain  this  to  software  people,  it  goes  over 
their  heads.  They  do  not  understand  that  when  you  are  making  a 
change  to  courseware,  you  may  have  to  change  something  in  the  TMS 
data  base.  You  have  to  track  the  data,  and  there  are  people 
assigned  to  do  just  that.  Anyway,  to  reiterate,  the  most  important 
report  that  we  receive  is  the  student  training  report,  and  it  must 
be  checked  very,  very  carefully.  Sometimes  when  you  got  these  long 
courses,  you  are  checking  484  objectives.  It  takes  hours,  but  you 
have  to  do  it. 

C llP_t  iquration  Management  Module 

In  every  other  system  that  I  visited,  the  configuration 
management  function  was  external,  but  in  the  C-130  program  it  is 
internal.  I  can  walk  in  to  the  contractor's  ottice  and  punch 
courseware  information  in  right  there  and  configure  it.  When  you 
realize  courseware  is  about  90%  of  our  effort,  the  timesaving  is 
enormous . 

When  we  do  a  design  request  and  assign  a  number,  we  go  right 
into  the  lesson  plan  configuration.  We  pull  out  the  lesson 
specification  which  identifies  all  of  the  lesson  segments,  frame 
references,  graphics  and  all  the  other  data  you  might  need  to  look 
at  that  might  need  to  be  changed. 
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since  it  is  internal,  it  helps  you  search  out  what  you  need 
faster,  including  CBT.  For  CBT,  you  still  have  to  go  to  a  CBT 
terminal  to  find  the  actual  material,  but  it  tells  you  where  to 
look.  You  can  code  it  in  a  variety  of  different  ways,  and  there 
are  different  ways  of  sorting  and  checking,  but  the  key  is  it  is 
all  internal. 

We  use  the  Informix  Relational  Database,  and  it  contains  data 
right  down  to  the  detailed  tables  so  information  can  be  tracked 
very  easily.  All  this  detail  is  just  beautiful  from  a  management 
point  of  view.  You  can  see  what  we  are  training,  where  we  are 
going,  and  where  the  gaps  are. 

Logistics  Management  Module 

The  last  module,  logistics  management,  does  not  have  to 
interface  with  the  other  modules.  It  is  strictly  an  off-the-shelf 
system  that  runs  your  maintenance,  logistics,  shipping  and 
receiving  for  the  depot  and  the  sites.  It  also  tracks  the  backlog 
of  work  orders  at  the  depot  and  all  of  the  sites.  It  tracks  all  of 
the  AWMs  (awaiting  maintenance)  on  a  daily  basis  so  you  do  not  get 
behind.  It  tracks  your  "awaiting  parts,"  so  you  know  if  you  have 
a  parts  problems  and  supply  problems,  etc.  The  information  is 
provided  on  management  charts  with  bar  codes  so  you  can  see  month- 
to-month  whether  we  are  getting  better,  i.e.,  this  many  work  orders 
open,  this  many  closed,  this  many  backlogged.  If  the  backlog  is 
getting  bigger,  then  you  know  you  have  problems  to  work  through. 

Test  and  Evaluation 


Without  T&E,  in  my  view,  neither  the  contractor  nor  the 
government  know  where  they  are  functionally.  They  would  not  have 
good  data  to  make  sound  decisions  about  where  they  are,  where  they 
are  going,  what  they  need,  and  how  well  the  system  is  working.  It 
is  not  so  much  a  T&E  job  to  say,  "Is  something  in  compliance  with 
the  contract?"  That  is  more  of  a  QA  function.  The  more  important 
issue  is — Does  it  work?  Is  it  operational?  Is  it  functional?  You 
can  build  an  ATS  that  is  totally  "in  compliance"  with  the 
specification  which  does  not  work  at  all.  "I  know  you  can  do 
that."  We  have  come  close  here  a  couple  of  times  with  aircraft 
schedules  and  operational  input  that  we  just  did  not  consider.  So 
T&E  in  my  view  is  absolutely  mandatory.  Of  course,  our  big  lesson 
learned  is  that  we  did  not  start  our  T&E  program  early  enough.  We 
are  doing  some  things  early  in  the  summative  evaluation  phase  that 
we  should  have  been  doing  in  formative  evaluation.  It  took  us  nine 
months  to  a  year  to  get  the  T&E  program  on  line,  and  the  contractor 
is  paying  the  price  now. 

A  good  system  level  formative  evaluation — "system"  is  the  key 
word  here — would  have  really  let  us  fix  many  things  early,  instead 
of  coming  back  aftei  SGTOs  and  having  a  lot  of  write-ups.  This  is 
tremendously  important  if  the  command  is  to  have  an  ATS  that  works 
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the  first  time.  This  is  a  critical  point.  The  traditional  T&E 
concept  is  that  the  government  can  allow  the  contractor  to  fail. 
"You  have  to  give  him  enough  rope  to  hang  himself."  In  ATSs,  you 
cannot  do  that.  You  cannot  stand  the  ATS  in  the  corner  and  say, 
"Well,  we  won't  use  the  simulator,  we'll  do  it  in  the  airplane;  or 
we  won't  use  the  CBT,  we'll  do  it  in  the  lecture."  ATSs  do  not 
work  that  way.  They  are  services  contracts,  they  are  not  hardware, 
and  a  hardware  contractual  mentality  will  kill  you.  It  has  to  work 
the  first  time  out  of  the  chute.  My  job  is  to  ensure  that  we 
graduate  a  student  that  meets  all  the  minimum  reguirements .  You 
have  to  answer  the  question,  "Can  he  do  the  job  coming  out  of  the 
chute  the  first  time?"  You  then  have  eight  months  of  summative 
evaluation  to  tweak  the  course  and  really  put  out  a  refined 
product . 

We  cannot  afford  to  let  the  contractor  fail.  Why?  Because 
our  resources  are  gone.  Our  expertise  is  gone  to  the  contractor. 
He  can  hire  our  blue-suit  people,  but  it  will  take  an  act  of 
Congress  to  get  that  guy  back  in  the  service  and  put  him  back  into 
the  blue  suit.  To  put  another  blue-suit  schoolhouse  together  would 
take  two  or  three  years  minimum,  in  my  view.  So  T&E  is  not  a 
subject  that  should  be  taken  lightly.  This  contractor  will  be  the 
first  to  admit  that  they  did  not  realize  the  value  of  it.  Now 
their  management  decisions  are  all  made  on  the  basis  of  T&E 
information.  In  the  past  they  would  say,  "We  just  don't  have 
enough  guys  to  go  around  for  T&E."  Now  they  have  hard  data  from 
T&E  to  pinpoint  where  and  how  effectively  their  resources  are  being 
used. 


Our  basic  T&E  master  plan  covers  the  formative,  summative,  and 
operational  evaluations  and  the  relationships  among  them.  It 
identifies  the  scheduled  start  and  end  points,  and  the  kinds  of 
activities  that  will  occur.  Any  ATS  should  have  this  comprehensive 
plan . 

Formative  Evaluation 


The  approach  to  formative  evaluation  in  this  program  is 
unique,  that  is,  we  divided  formative  evaluation  into  subordinate 
plans.  There  are  formative  evaluation  subordinate  test  plans  for 
ISD  and  the  TMS.  The  Training  System  Support  Center  (TSSC)  is  a 
real  service-oriented  kind  of  test  situation.  It  tests  the 
contents  in  your  CLS.  We  have  also  started  qetting  into 
configuration  kinds  of  issues  in  most  tests,  and  the  main  thing  is 
operation  of  the  Configuration  Working  Group  (CWG) .  When  you  start 
getting  into  the  day-to-day  operations,  how  do  you  pass  the  paper 
along?  What  kind  of  maintenance  are  you  going  to  do  to  keep  your 
people  current?  It  should  all  be  tested  in  formative  evaluation. 
This  applies  to  the  point  I  made  earlier  about  guaranteeing  the 
" factory. " 
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System  integration  testing  has  been  a  big  problem  for  us  in 
formative  evaluation.  In  fact,  it  has  probably  slipped  over  into 
summative  by  default.  We  fought  very,  very  hard  to  get  them  to 
test  the  TMS  to  some  level  for  integration  during  SGTOs.  In  the 
first  three  blocks,  the  TMS  was  not  ready.  In  retrospect,  the 
government  should  have  said,  "Sorry  contractor,  you  don't  run  your 
SGTO  until  the  TMS  is  operating."  What  happened  was  that  the  TMS 
was  a  separate  contract  line  item,  so  the  software  folks  were 
operating  on  a  schedule  to  meet  that  requirement  alone.  They  were 
not  operating  under  a  network  umbrella  to  ensure  that  the  TMS  along 
with  the  courseware,  instructors,  and  facilities  all  came  together 
at  SGTOs.  So  systems  integration  was  a  problem  for  us;  it  still 
is.  We  are  picking  up  the  ball  and  running  with  it  in  the  early 
classes  of  summative  and  we  are  getting  it  done.  But  it  is  costing 
the  contractor  more  dollars. 

Summative  Evaluation 


The  two  key  contractor  people  responsible  for  the  summative 
evaluation  phase  previously  worked  on  the  C-5  ATS  T&E  program.  As 
a  consequence,  they  were  able  to  apply  all  of  their  lessons 
learned  and  experience  to  this  program.  They  never  had  a  quality 
T&E  report  out  of  the  C-5;  the  approach  and  procedures  they  have 
developed  for  this  program  have  produced  a  tremendous  document.  In 
fact,  it  covers  everything  from  student  critiques,  to  scheduling, 
TMS  support,  instructor  critiques,  supervisor  evaluations,  you  name 
it.  For  instance,  we  are  obtaining  detailed  course  critique  data 
from  about  20%  of  the  students  going  through  the  courses.  That  is, 
we  use  20%  of  the  throughput  from  each  group  of  students  going 
through  a  course,  and  we  are  critiquing  all  of  the  lessons  in  each 
course . 

Operational  Evaluation 

The  operational  evaluation  plan  is  not  due  until  30  days 
before  TSRR.  Although  I  have  not  seen  a  draft  because  it  is  too 
early,  I  feel  it  will  be  a  good  plan.  I  hope  to  take  the  best  of 
the  summative  evaluation,  the  best  lessons  learned,  and  also  the 
best  information  out  of  the  Management  Indicators  report  and 
continue  those  procedures  for  the  life  of  the  program.  I  think 
that  is  a  sound  approach. 

A  final  comment  about  T&E.  When  I  talked  to  the  SOF  ATS 
contractors,  it  was  obvious  that  T&E  to  them  is  hardware.  So  they 
are  talking  DT&E  (Development  Test  and  Evaluation) ,  OT&E,  48-hour 
requirement  on  software  and  that  sort  of  thing,  and  the  bigger 
issues  that  are  going  to  sink  the  ship  at  the  program  level  do  not 
even  cross  their  minds. 
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Quality  Assurance 

Quality  assurance  (QA)  is  one  of  my  biggest  lessons  learned 
and  one  of  the  things  I  am  proudest  of.  In  my  view,  it  is  second 
in  importance  only  to  T&E.  In  trying  to  get  a  handle  on  this  area 
during  source  selection,  I  found  out  that  the  manufacturing  people 
at  ASD  did  not  want  any  part  of  it,  AFOTEC  (Air  Force  Operational 
Test  and  Evaluation  Center)  did  not  want  any  part  of  it  either,  nor 
did  the  test  people  at  Headquarters  MAC.  Therefore,  it  was  left  to 
us  to  handle  the  "quality”  and  T&E  functions.  We  built  that  part 
of  the  program  from  the  ground  up.  It  is  a  unique  philosophy,  but 
I  will  say  that  we  have  done  it  in  close  coordination  with  AFCMC 
(Air  Force  Contract  Maintenance  Center)  and  received  their  guidance 
to  make  sure  we  are  in  accordance  with  Air  Force  regulations,  par¬ 
ticularly  AFR  74-15  (Procurement  Quality  Assurance) ,  which  is  the 
government  QA  bible. 

The  major  difficulty  for  us  was  overcoming  the  hardware 
orientation  of  the  traditional  QA  program.  The  acquisition 
community  is  set  up  to  handle  hardware.  In  fact,  this  was  the 
first  contract  to  come  out  of  ASD  that  had  a  clause  for  a  "services 
oriented"  QA.  When  ASD  found  out  about  that,  they  wanted  to  take 
it  out  because  they  did  not  know  how  to  manage  it.  What  has  really 
paid  dividends  for  us  is  having  two  full-time  GS-12  QARs  (Quality 
Assurance  Representatives)  and  one  major  who  had  the  capability  to 
build  a  QA  program  from  the  ground  up  that  covers  all  aspects  of 
the  ATS.  It  covers  training  operations,  courseware,  TMS ,  TSSC,  and 
all  those  areas  that  have  not  been  a  part  of  the  traditional 
hardware/software  QA  programs. 

Contractor  Quality  Assurance  (QA) 

Quality  assurance  for  the  contractor  is  basically  just 
ensuring  that  he  is  complying  with  his  own  policies,  procedures, 
processes,  and  plans  in  a  best-commercial-practice  manner  in  a 
situation  where  you  are  allowing  him  to  bring  his  innovative  skills 
to  the  table.  In  some  cases  we  care,  but  in  most  cases  we  do  not 
care  how  he  runs  his  program.  What  we  do  care  about,  though,  is 
our  desire  to  maintain  the  factory  guarantee  at  some  sort  of  a 
standard,  and  that  standard  is  whatever  the  contractor  defines  as 
the  standard.  What  we  have  found  in  our  investigation  of  other 
ATSs  is  that  if  you  do  not  monitor  the  program  (particularly  after 
contract  award) ,  the  standard  tends  to  drift  down  and  the  tendency 
is  to  lean  out,  lean  out,  and  lean  out  to  increase  the  profit 
margin.  So  you  need  some  way,  acceptable  to  both  sides,  to  ensure 
that  what  you  bought  is  what  you  maintain  so  that  the  government  is 
positioned  for  recompetition  should  it  become  necessary.  By  the 
way,  except  for  the  two  new  ones — C-141  and  C-17--there  is  not  an 
ATS  out  there  that  still  has  the  same  contractor  who  won  the 
original  award.  They  have  all  been  sold  off  or,  like  the  E-6,  have 
been  terminated  and  recompeted.  You  have  to  plan  for  recompetition 
to  protect  the  government.  Either  that  or  you  have  a  factory  that 
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nobody  wants  to  take  over  without  a  large  amount  of  additional 
funding.  In  some  cases,  you  may  have  to  recompete  without  much 
prior  warning  when  a  contractor  decides  that  he  wants  to  get  out  of 
the  military  training  business. 

Another  important  point  that  I  want  to  make  about  QA  is  that 
it  is  an  external  function;  that  is,  it  has  to  be  independent  of 
the  areas  that  it  is  monitoring.  It  should  be  external  for  both 
the  contractor  and  the  government.  One  of  our  problems  early  on 
was  that  the  contractor  always  wanted  their  QA  people  to  work 
internally  within  the  divisions.  We  ran  into  some  big  problems 
because  when  they  would  find  they  were  not  in  compliance,  their  own 
managers  would  not  let  them  write  it  up,  and  it  just  cannot  work 
that  way.  This  approach  to  me  is  QC,  not  QA.  We  finally  reworked 
that  issue,  and  they  now  have  an  independent  QA  staff  of  four  full¬ 
time  people — a  QA  manager  running  it,  a  software  person,  a 
courseware  person,  and  a  maintenance/logistics  person.  Right  now 
they  probably  need  another  person  to  handle  the  training  operations 
side,  since  most  of  our  current  problems  involve  the  integration 
between  the  flight  line  and  the  contractor.  These  include  such 
things  as  signing  off  check  rides  and  remediation  issues  that  do 
not  really  fall  into  those  other  three  areas. 

The  big  problem  for  contractors  in  QA  is  that  they  will  have 
a  QA  policy  for  hardware  and  software,  but  not  for  ATSs.  I  have 
been  around  talking  to  other  vendors  such  as  Hughes,  McDonnell 
Douglas,  Logicon,  and  General  Electric,  and  none  of  them  have  QA 
policies  and  procedures  specifically  designed  for  training  systems. 
QA  policy  is  generated  at  the  corporate  level  as  in  TQM  (Total 
Quality  Management)  which  emanates  from  that  corporate  level 
president.  You  can  call  Link  in  Binghamton,  NY,  and  find  someone 
who  works  for  the  company  president  who  is  in  charge  of  QA,  but  the 
first  thing  he  tells  you  is,  "I  only  do  hardware."  If  you  ask, 
"Who  does  courseware?"  or,  "Who  does  your  instructor  hiring  and 
maintenance  to  make  sure  they  maintain  their  skills?"  or,  "Who's 
responsible  for  the  TMS  and  TMS  software  and  TMS  data?"  and  there 
are  no  answers. 

The  first  thing  we  had  to  do  was  go  back — and  it  took  months 
to  do  this — get  a  policy  written  and  then  get  the  president  to  sign 
off  on  it.  Once  there  was  a  policy,  the  next  step  was  for  each 
division  to  write  a  division  level  manual  to  implement  that  policy. 
Neither  of  those  documents  are  specific  to  the  C-130.  They  are  for 
ATSs  in  general,  but  they  are  systematic.  After  each  division  has 
its  manual,  the  contractor  QA  staff  can  write  the  QAPP  (Quality 
Assurance  Program  Plan)  which  is  usually  a  contractual  deliverable. 
Of  course,  the  local  group  may  write  a  plan  and  say,  "Here's  how." 
Then  the  division  will  say,  "No,  that's  not  how  we  want  to  manage 
that."  So  they  will  have  to  go  back  to  be  compliant  with  the 
management  procedures — who  reports  to  whom,  or  whatever. 
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QDIs  (Quality  Departmental  Instructions)  are  what  make  up  most 
of  the  QAPP.  There  are  some  general  statements  about  how  you  are 
going  to  build  it  and  what  it  is  going  to  cover;  then  you  get  down 
to  the  QDIs,  which  are  actually  just  checklists.  You  should  have 
a  checklist  for  just  about  every  procedure,  every  process,  every 
plan,  everything  that  you  are  doing.  QDI  is  an  external  checklist 
that  the  Quality  Assurance  Department  uses  for  their  random  checks 
(approximately  5%)  every  six  months. 

QDIs  are  based  on  DWIs  (Departmental  Work  Instructions) .  What 
does  that  mean?  For  example,  in  the  formative  evaluation  ISD 
subordinate  test  plan,  they  explain  in  great  aetail  these  55  steps 
for  developing  a  lesson.  For  each  step  they  say,  "Our  department 
will  do  this,  QC  will  do  this,  and  Instructional  Development  (ID) 
will  do  this  on  each  lesson."  They  also  explain  for  each  group 
that  when  they  review  the  lesson  process,  they  will  look  for 
this...  and  so  on.  Those  are  the  work  instructions,  and  this  is 
how  they  are  going  to  develop  lessons.  If  we  accomplish  all  of  the 
55  steps,  thei.  I  feel  comfortable  that  when  a  lesson  comes  out  at 
the  end,  after  all  of  the  T&E  and  government  checks,  it  will  be  a 
good  lesson. 

A  couple  of  little  problems  we  have  had  that  make  good  lessons 
learned  follow. 

Let  us  say  you  do  not  like  your  55  steps,  you  want  to  make  it 
49  because  you  have  learned  some  things  and  are  a  little  more 
efficient.  For  instance,  you  do  not  need  the  art  department  to 
review  simulator  instructor  guides,  since  there  is  no  art  work.  So 
it  makes  sense  to  cut  that  step  out.  However,  if  you  do  not  tell 
your  quality  people,  both  in  the  government  and  in  your  own 
company,  you  are  changing  the  procedure,  then  when  they  come  over 
to  do  an  inspection,  they  do  not  care  that  there  is  no  artwork.  It 
makes  sense  for  the  lesson  not  to  go  to  art,  but  your  procedure 
says  it  has  to  go  to  art.  You  get  a  write-up  because  you  are  not 
following  your  own  procedure.  We  have  had  to  teach  them  that  it  is 
fine  to  change  it,  we  want  you  to  change  it,  we  want  you  to  be  more 
efficient,  we  want  you  to  make  a  profit,  but  change  it  through  the 
approval  cycle.  First,  get  the  change  approved  and  then  implement 
it.  Do  not  just  implement  it  willy-nilly,  and  most  of  our  QDRs 
were  just  that.  There  were  better  ideas,  but  they  were  not 
following  proper  channels  for  changing  procedures. 

Changes  in  procedures  have  to  be  controlled.  Everybody  cannot 
do  what  he  thinks  is  best  without  formal  coordination.  There  are 
a  lot  of  problems  between  sites  in  that  regard  with  respect  to  such 
things  as  the  maintenance  support  plan,  logistics  support  plan,  and 
so  on.  Each  site  manager  could  say,  "Oh  no,  I  don't  like  that. 
Here's  how  I'm  going  to  do  it."  Well,  our  guys  would  go  out  and  do 
a  QA  visit  and  say,  "Hey,  you're  not  in  compliance  with  the  plan." 
"Oh  I  know.  I  got  a  better  idea."  Wrong,  sorry!  Get  them  to 
change  it  at  the  program  level.  Because  we  want  an  audit  trail,  we 
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want  a  factory  out  there  that  is  standardized,  so  that  we  can  sell 
it  to  another  contractor  if  required. 

Government  OA 

Despite  the  fact  that  QA  is  really  a  contractor 
responsibility,  you  must  have  on-site  government  surveillance  to 
ensure  that  it  gets  done  properly.  Although  both  the  C-17  program 
and  the  C-141  program  have  been  in  being  for  some  time,  there  is 
still  no  on-site  contractor  QA.  I  understand  they  are  trying  to  do 
QA  for  the  C-17  out  of  Long  Beach,  CA.  They  recognize  now  they 
cannot  do  that  and  are  trying  to  hire  somebody  to  put  on  that 
position  at  Norman,  OK.  The  key  here  is  to  require  external  QA 
personnel  working  for  program  management  at  the  individual  program 
level  as  well  as  corporate. 

You  cannot  write  the  government  QA  plan  until  you  have  the 
QDIs  from  the  contractor's  QAPP.  We  use  Formtool  with  two  side-by- 
side  windows.  On  one  side  is  the  QDI  checklist  and  the  other  side 
is  what  we  will  check  to  verify  a  QDI  checklist.  Primarily,  what 
we  do  is  look  at  their  quality  reports  to  make  sure  that  they  do 
what  they  said  they  were  going  to  do.  If  they  say  they  are  going 
out  every  three  months  to  look  at  something,  are  they  actually 
going  out  to  look  at  it  and  what  do  their  reports  reflect?  Also, 
are  they  doing  follow-ups?  If  we  go  out  and  do  our  checks,  we 
exclusively  use  their  checklists  so  there  are  no  surprises.  The 
other  nice  thing  about  QA  is  that  not  only  are  there  no  surprises, 
but  most  of  the  issues  that  come  up  we  try  to  solve  at  a  low  level. 
If  it  does  not  get  solved  at  the  worker  level,  I  throw  it  in  the  QA 
arena  and  have  their  QA  and  my  QA  get  in  the  books,  get  in  the 
proposal,  get  into  the  plans  and  make  a  decision.  In  almost  all 
cases,  they  work  it  out.  If  they  agree,  then  we  do  not  write  a 
QDR,  Method  B  or  anything.  We  give  them  time  to  work  it,  and  if  at 
some  point  they  are  not  in  compliance  and  they  say,  "Yes,  we  know 
it,  but  we're  not  going  to  do  anything  about  it,"  that  is  when  we 
formally  write  them  up.  We  give  them  every  opportunity  to  fix  it, 
and  that  kind  of  relationship  has  really  made  QA  work  very  well. 

QA  is  not  a  black-and-white  issue  in  an  ATS  as  it  is  for 
hardware  and  software.  You  cannot  put  calipers  on  students  and  you 
do  not  hook  an  oscilloscope  to  their  heads  to  measure  what  they 
have  learned.  After  all  of  our  formative  and  summative  evaluation, 
you  have  to  build  a  program  that  is  operationally  feasible — and  it 
works.  So  that  is  what  we  are  "QA-ing"  against.  You  work  out  the 
differences  on  the  plans. 

Another  area  in  QA  is  ATD  training.  Simcert  (the  Air  Force 
simulator  certification  group)  is  a  big  part  of  it;  they  are 
involved  in  QA  with  us  on  the  hardware  and  the  fidelity  checks. 
CLS  is  also  a  big  part  of  it.  Of  course,  the  bottom  line  is  that 
not  only  are  we  training  and  have  devices  to  support  the  training 
and  are  getting  that  guaranteed  student  against  all  of  those 
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objectives,  but  we  are  involved  in  QA — and  this  is  probably  the 
biggest  part — we  QA  to  ensure  quality  and  to  make  sure  we  have  a 
viable  recompetition  package. 

Let  me  give  you  an  example.  One  year  into  the  program  I 
asked,  "Is  every  item  officially  tagged  that  we  (the  Air  Force) 
bought  under  contract  for  this  program?  Or  when  they  finish 
development,  are  they  going  to  haul  a  bunch  of  our 
equipment/furniture  away?"  We  called  the  C-5  guys  to  see  how  they 
identified  all  of  their  equipment.  We  called  them  because  they 
were  two  years  ahead  of  us,  and  they  said,  "Oh,  you  know  we  haven't 
tagged  anything  either."  So,  we  went  into  a  big  effort  with  an 
inventory  sheet  to  tag  and  put  stickers  on  all  the  items  the 
government  bought.  That  is  all  the  items  that  stay  if  the 
contractor  should  ever  leave.  The  contractor  also  purchased  some 
additional  items  on  their  own.  This  whole  issue  is  very  important 
to  consider  for  recompetition,  the  furniture,  typewriters,  etc., 
that  we  purchased  have  to  stay.  If  they  take  all  of  the  copy 
machines  and  all  of  the  computers,  the  next  contractor  that  comes 
in  has  nothing  to  work  with,  and  it  is  going  to  be  expensive  for 
the  government. 


Configuration  Management 

Configuration  management  includes  the  standard  hardware  and 
software  functions  which  have  been  done  for  years  and  are  pretty 
cut  and  dried.  When  you  ask  for  "best  commercial  practice,"  the 
contractors  all  tend  to  follow  AFR  57-4  (Modification,  Approval  and 
Management) ,  because  that  is  what  they  have  all  used  in  the  past. 
The  hardware  arena  has  not  been  a  problem  on  this  program  and 
probably  will  never  be  a  problem  except  for  the  integration  with 
the  rest  of  the  system.  The  TMS  software,  however,  is  another 
matter.  There  is  no  set  standard  established  for  computer 
software,  and  there  are  many  for  "best  commercial  practices."  Our 
contractor  chose  to  use  DOD-STD-2167  (Defense  System  Software 
Development),  but  we  had  to  tailor  it  drastically  to  fit  our  needs. 
It  has  traditionally  been  used  to  write  software  for  simulators. 
We  downgraded  the  requirements  to  six  DIDs  (Data  Item  Descriptions) 
that  described  such  things  as  the  software  test  procedures, 
software  test  reports,  etc.  I  cannot  remember  the  others,  but 
there  were  six  DIDs  used,  and  it  has  worked  very  well  for  us.  It  is 
very  important  to  have  a  standard  of  some  kind  so  that  it  will  be 
configured  and  baselined  so  that  we  could  QA  it  in  accordance  with 
"something."  Since  the  TMS  itself  was  being  developed  in  Dallas, 
TX,  we  tasked  DCAS  (Defense  Contract  Administration  Services)  in 
Fort  Worth  to  QA  the  TMS  programming  development.  They  had  no  idea 
what  was  going  on,  had  never  done  this  type  of  oversight  before, 
and  we  have  not  received  very  much  input  from  them.  This  could 
lead  to  some  problems  down  the  line. 
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Courseware 


Courseware  is  a  new  and  very  complex  issue  for  the  acquisition 
community.  My  view  of  this  is  that  a  contractor  probably  needs  to 
bring  at  least  one  good  expert  onboard  very  early  to  set  up  the 
development  program.  You  .  eally  do  not  get  involved  in  this  until 
those  first  lessons  start  reaching  the  last  five  or  six  steps  in 
the  55-step  process.  But  once  that  happens,  the  volume  is  awesome. 
The  contractor  has  fifteen  full-time  people  working  changes,  and  if 
you  walk  across  the  street,  there  are  mounds  of  documents.  I  think 
the  last  I  saw  there  were  350  DRs  open  that  they  were  trying  to 
accomplish  with  fifteen  people.  It  is  a  very  labor-intensive 
thing,  but  it  is  a  unique  process.  I  must  note  that  in  the 
courseware  area,  it  is  not  only  the  courseware  lessons.  Included 
are  the  task  listings,  student  orientation  manuals,  and  course 
summary  documents.  All  of  these  must  be  kept  current  with  changes 
to  the  lesson;  you  cannot  concentrate  on  the  lesson  only.  Because 
of  the  continuing  volume  on  a  program  this  big,  you  will  never 
catch  up  if  you  get  behind  on  the  supporting  ISD  documents. 

TMS  Data  Configuration 

Interposed  between  the  TMS  software  and  the  courseware  is  the 
TMS  data.  The  issue  here  is  who  owns  the  data,  and  how  do  you 
configure  it?  In  the  courseware  area,  there  are  always  many 
changes  to  test  questions,  e.g.,  rewriting  bad  questions  and 
rephrasing  the  questions  for  foreign  students,  etc.  This  usually 
generates  changes  to  the  TMS  database,  such  as  the  grading 
procedure  in  the  performance  measurement  module  which  must  be  kept 
concurrent.  We  became  aware  of  the  need  for  configuration 
management  of  this  area  when  we  would  change  a  test  and  hand  it  out 
to  the  student.  The  student  would  take  the  test  and  get  an  answer 
right,  but  the  TMS  would  mark  him  wrong.  We  would  remediate  and  he 
would  say,  "Wait,  I  got  that  right,"  and  upon  checking,  we  found 
out  that  he  did  get  it  right. 

Integrated  Configuration 

The  integrated  configuration  area  is  very  difficult 
conceptually.  For  instance,  you  have  to  implement  the  software 
changes  and  data  changes  at  the  same  time  that  the  instructor  hands 
out  the  new  courseware,  and  that  is  very  important.  I  will  give 
you  a  good  example;  When  I  sat  on  the  configuration  working  group, 
and  someone  brought  up  a  subject  for  consideration,  the  first  thing 
I  would  ask  myself  is,  "Does  this  DR  (Deficiency  Report)  affect 
anything  else  in  the  system?"  If  you  do  not  think  like  that,  you 
will  get  noncurrent  real  quick. 

For  example,  we  had  a  TO  (technical  order)  change  that  changed 
the  location  of  the  lights  on  drop  zones  for  night  drops.  A  DR  was 
written  to  change  the  visual  system,  which  was  approved.  Then  I 
started  thinking  and  said,  "Hey,  is  there  any  academic  courseware 
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that  tells  the  navigators  or  anyone  else  what  to  look  for  prior  to 
the  simulator  lesson?"  ISD  said,  "Yes,  that's  in  a  lesson."  I 
then  asked,  "Do  you  think  we  ought  to  write  a  DR  for  the  academic 
lesson  too?"  This  approach  prevents  students  saying,  "That's  not 
what  they  taught  me  in  academics."  We  did  more  research  and  found 
out  there  was  a  test  question  for  the  navigators  that  covered  where 
the  lights  are  located  for  night  drops.  We  had  to  go  back  to 
change  the  test  question  which  was  in  the  TMS  data  base.  We  ended 
up  with  three  DRs  on  one  subject  at  the  same  time. 

In  an  ATS  environment,  you  simply  cannot  follow  the 
traditional  hardware/software  configuration  approach.  For  example, 
you  cannot  simply  write  DRs  to  take  care  of  simulators,  because 
they  impact  other  parts  of  the  system,  particularly  the  TMS  and 
courseware.  We  have  refined  and  integrated  the  process,  and  it 
works  very  smoothly.  During  the  assessment  of  a  DR,  we  actually 
have  boxes  that  we  check  off  that  say,  "Does  this  affect  the  TMS? 
Does  this  affect  the  lesson  specification?  Does  it  affect  the 
MSSR?"  If  it  does,  you  attach  the  appropriate  sheet,  and  then  it 
is  logged  into  the  configuration  module.  It  does  not  get  the  final 
signoff  unless  all  of  the  other  areas  are  completed  and  signed  orr 
also.  This  is  a  very  important  concept  in  an  ATS,  and  that  is  why 
it  takes  fifteen  people.  It  is  not  just  looking  at  a  lesson  that 
has  a  few  sentences  changed;  it  is  the  assessment  and  checking  to 
see  what  else  it  affects  across  the  system. 

Configuration  Working  Group 

The  Configuration  Working  Group  (CWG)  is  where  it  all  gets 
managed.  The  usual  trigger  is  a  write-up  emanating  from  the  T&E 
process,  except  for  a  formal  modification  to  simulator  hardware  or 
software.  For  example,  any  courseware  write-up  or  any  TMS  write-up 
would  start  out  as  an  ICP  (instructional  change  proposal)  in  the 
T&E  arena.  However,  we  are  not  limited  to  courseware  only,  it  can 
be  on  anything  in  the  system.  The  flight  ]  ine  instructors  or 
anyone  else  who  sees  anything  can  submit  a  write-up.  The  write-up 
goes  through  an  analysis  stage  and  is  given  a  number.  It  comes 
down  to  my  folks  for  review  and  when  both  sides  agree  that  it  is 
valid,  it  is  submitted  to  the  CWG  and  turned  into  a  DR.  The  CWG 
prioritizes  the  DRs  as  critical ,  significant ,  or  routine .  The 
preponderance  of  what  we  have,  the  important  ones,  are  usually 
categorized  as  significant.  We  have  not  had  any  crit icals .  The 
criticals  are  safety-of-f light  and  have  to  be  implemented  within  24 
hours . 

We  have  had  quite  a  few  significants--? 50 .  I  think — that  we 
have  been  running  for  the  last  couple  of  months  and  a  lot  more 
routines .  Routines  are  quarterly  updates,  typos,  references,  and 
items  like  that.  The  members  of  the  working  group  for  the 
government  include  government  QA  people.  That  is  to  make  sure  the 
proper  process  is  followed.  When  we  close  out  each  DR,  the  final 
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step  is  QA's  stamp  that  certifies,  "We've  gone  through  all  of  the 
hoops  and  closed  it  out  as  an  official  change  to  the  baseline." 

Simcert  also  sits  in  on  the  CWG.  They  are  our  experts  on 
what  is  going  into  the  simulator,  and  they  are  in  the  best  position 
to  judge  whether  or  not  suggested  simulator  changes  are  valid,  nice 
to  have,  or  hokey.  This  oversight  is  important  to  keep  costs  down. 
I,  myself,  have  been  representing  operations.  But  if  we  have  a  lot 
of  DRs  for  navigators,  for  example,  I  will  take  a  navigator  along  or 
whatever  other  kind  of  expertise  is  required  by  the  agenda.  The 
contractor  also  will  bring  in  their  experts  to  discuss  items  when 
necessary.  For  the  most  part,  I  do  not  question  the  individual 
items  since  the  write-ups  have  been  reviewed  and  proofed  by  the 
government  SMEs  as  ICPs.  When  I  do  question  something,  it  is 
usually  on  the  potential  impact  relative  to  other  parts  of  the 
system. 


The  contractor's  configuration  manager  runs  the  CWG  meeting 
and  has  the  final  say  on  priorities  and  so  forth  because  it  is 
their  system.  We  can  still  disagree  and  the  next  step  is  the  SRB 
(System  Review  Board)  which  is  run  out  of  HQ  MAC/DOT.  Generally, 
however,  it  has  not  been  a  problem  coming  to  an  agreement.  The 
first  meeting  we  ran  went  four  hours.  We  had  a  lot  of  non- 
essential  people  at  the  meeting,  and  they  were  fighting  among 
themselves.  To  correct  this,  we  decided  to  build  the  agendas  ahead 
of  time  and  get  them  out  the  day  before. 

Lessons  Learned 

At  this  point,  I  will  attempt  to  summarize  the  major  lessons 
learned  from  my  experiences  in  this  program.  Some  of  these  lessons 
tie  into  things  we  have  already  talked  about.  However,  there  is 
another  major  area  which  I  have  not  covered,  but  should  be  a  major 
consideration  in  all  ATS  programs.  It  is  the  need  to  ensure  that 
the  system  is  designed  to  interface  and  function  effectively  within 
the  context  of  all  the  relevant  government  regulations, 
communication  channels,  etc.,  that  are  a  part  of  the  program's 
operating  environment  within  the  military. 

Spend  Time  on  RFP 

When  the  contractor  gets  ready  to  develop  and  implement  an 
ATS,  every  sentence  in  the  RFP  (Request  for  Proposal)  is  looked  at 
with  a  microscope.  In  the  C-130  we  probably  spent  more  time  than 
any  other  ATS  program  fighting  hard  for  operational  issues.  Even 
that  in  my  view  was  not  enough.  If  I  had  it  to  do  over,  there  are 
some  areas  that  I  would  ensure  that  more  consideration  be  given  to 
operational  issues  which  cannot  be  well  defined  in  a  statement  of 
work.  We  recognize  that  the  contractor  wants  as  much  specificity 
as  possible  so  he  knows  what  he  is  bidding  on.  However,  there  are 
many  times  you  do  not  really  find  out  what  it  takes  to  develop  and 
field  an  ATS  until  you  are  at  an  operational  site  and  become 
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familiar  with  the  operational  requirements  and  constraints.  Much 
of  the  precontractual  work  is  done  in  a  relative  vacuum  in  this 
area . 

Check  CLINS  Against  the  Winning  Proposal 

You  should  do  a  thorough  postaward  audit  to  ensure  that  the 
contractor's  proposal  satisfies  the  real  requirements  of  all  of 
your  contract  line  items  (CLINS) .  There  are  a  number  of  areas 
where  we  had  problems  because  we  missed  some  important  disconnects 
between  the  proposal  and  real-world  requirements.  One  of  the 
crucial  ones  was  the  TMS.  It  was  out  there  all  by  itself  and  not 
required  contractually  to  come  on  line  with  the  courses.  These  two 
requirements  should  have  been  tied  together.  Another  one  was  that 
our  SOW  (Statement  of  Work)  had  one  CRR  at  the  end  of  28  months, 
and  the  contractor  elected  to  do  it  in  nine  separate  blocks,  which 
was  a  lot  smarter.  However,  if  they  did  not  make  their  milestones, 
we  could  not  do  anything  to  them  for  changing  them  around  because 
there  is  no  penalty,  there  are  no  incentives,  there  is  nothing 
there  because  we  never  changed  the  contract  to  match  the  nine 
projected  CRRs.  I  would  recommend  that  as  soon  as  you  pick  a 
winner,  use  the  postaward  conference  to  go  back  through  the 
contract  and  look  at  what  is  in  there.  Make  sure  that  there  is  a 
proper  match  and  identify  some  incentives  and  some  penalties  tied 
to  the  contractor's  performance. 

Require  a  Detailed  Network  Schedule 

A  detailed  network  schedule  is  critical,  especiallv  for  the 
acquisition  agency.  In  ray  view,  we  never  had  one  for  the  C-130 
except,  perhaps,  for  the  SCNS  modification.  We  reworked  the 
schedules  again  and  again  trying  to  get  all  of  the  program  elements 
integrated.  We  started  with  a  set  of  relatively  independent 
schedules.  For  instance,  the  TMS  tolxs  ouxlt  rjieir  development 
schedule  in  Dallas,  the  courseware  folks  built  theirs,  the  people 
that  were  hiring  the  instructors  had  theirs,  and  there  was  another 
for  facilities.  We  were  down  to  the  wire  getting  electricity  and 
air  conditioning  in  to  run  in  the  computer  rooms  in  time  for  the 
first  courses.  As  I  stated  before,  we  had  to  run  three  SGTOs  with 
no  TMS  because  nobody  had  looked  at  the  bigger  picture.  We  were 
delivering  lessons  to  SGTOs  one  day  before  they  were  taught. 
Nobody  had  thought  about  the  fact  that  the  instructors  needed  time 
to  study  before  they  met  their  students.  The  need  to  develop 
network  schedules  during  early  planning  is  critical  to  guarantee 
that  everything  comes  together  when  needed.  You  also  have  to 
integrate  your  testing.  When  do  these  test  points  fall  out  so  that 
you  know  the  course  has  a  chance  of  working  when  it  is  networked 
together? 
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start  T&E  and  QA  Early  On 


You  need  your  initial  T&E  and  QA  cadres  on-line  from  the 
beginning  because  they  have  to  start  working  on  the  timing, 
integration,  plans,  policies,  procedures,  processes,  to,  if  nothing 
else,  get  that  overall  big  umbrella.  It  starts  with  a  corporate 
policy  that  supports  the  program.  You  have  to  build  a  master  test 
plan  before  you  can  build  one  for  each  of  the  divisions.  You  have 
to  define  your  goals  and  objectives,  identify  the  working 
relationships,  etc. ,  and  then  you  can  develop  the  detailed 
procedures . 

Define  Goals  (CRR,  TSRR  Requirements^ 

With  respect  to  procedures,  CRRs  were  very  easy  for  us.  We 
had  a  28-page  checklist,  and  there  were  no  surprises.  If  you 
accomplished  everything  on  that  checklist,  then  a  government  review 
was  a  nonevent.  If  there  was  an  action  item,  everybody  knew  it  was 
an  action  item  before  the  meeting.  As  long  as  they  had  an  agreed- 
upon  date  to  have  the  action  item  cleaned  up,  CRRs  were  no  problem. 
The  same  thing  for  TSRR.  We  have  just  finished  a  quarterly 
Management  Indicator  report  that  displays  all  of  the  relevant 
system  trends.  If  we  are  within  the  goal  limits  and  all  issues  are 
cleaned  up,  along  with  all  of  the  ICPs  and  DRs,  it  will  be  a 
nonevent.  We  have  identified  and  agreed  upon  the  goals  and 
objectives  one  year  before  they  are  due.  There  should  not  be  any 
discussion  at  the  last  minute  about  what  we  thought  should  happen 
versus  what  did  happen,  and  it  really  should  be  easy.  Although 
what  we  are  doing  is  listed  as  basic  principles  in  all  of  the 
management  textbooks,  it  is  not  implemented  in  some  of  the  other 
ATS  programs  and  was  difficult  on  this  one. 

lia ke  Advantage  of  the  Current  System 

Contractors  should  take  advantage  of  what  is  already  available 
in  existing  systems  before  starting  to  build  a  new  one  from 
scratch.  We  probably  could  have  saved  a  lot  of  time  and  many,  many 
man  years  of  effort  if  the  contractor  had  just  looked  at  our 
existing  program  and  used  it  as  a  point  of  departure.  MAC  has  been 
training  C-130  aircrews  for  thirty  years,  so  there  is  a  lot  of 
wisdom  and  experience  that  has  been  incorporated  into  the  system. 
The  contractor  could  have  gotten  a  leg  up  on  a  number  of  issues 
involving  such  things  as  aircraft  generation,  instructor 
requirements,  facility  limitations,  etc.,  by  taking  advantage  of 
information  already  in  existence.  In  my  view,  they  could  have 
saved  a  tremendous  amount  of  time  and  effort  had  they  done  that, 
but  they  wanted  to  do  it  all  new  from  the  beginning.  That  was 
their  choice.  However,  they  ended  up  going  back  and  using  a  lot  of 
what  we  have  today,  particularly  in  the  tactical  area,  because  it 
is  so  complex  and  because  of  certain  limitations  on  aircraft 
generation  capability.  In  many  cases,  there  probably  is  not  a 
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significantly  better  way  to  do  some  of  the  things  being  done  in  the 
old  system  or  it  would  have  been  done  previously. 

Real-World  Reguirements 

When  a  conflict  occurs  over  real-world  requirements,  they  have 
to  take  precedence  over  the  SOW  and  the  specification.  If  you  have 
to  change  the  contract  to  meet  these  requirements,  then  you  change 
it.  A  good  example  is  the  concept  of  integrating  aircraft  sorties 
with  the  ground  training  portions  of  a  course.  This  concept 
initially  would  have  produced  a  ten-aircraft  generation  requirement 
surge  one  day  a  week.  I  went  down  to  the  DCM ' s  (Deputy  Commander 
for  Maintenance)  shop  to  discuss  it,  and  he  laughed  as  he  was 
throwing  me  out  of  his  office.  He  said,  "Dream  on.  I  can't 
generate  airplanes  like  that."  There  is  nothing  in  the  RFP  or  SOW 
that  says  the  contractor  has  to  talk  to  the  DCM  to  see  what  he  can 
generate,  but  you  still  cannot  build  these  requirements  in  a 
vacuum.  They  have  to  be  implemented  in  the  real  world.  These 
real-world  requirements  are  a  fact  of  life  and  you  usually  don't 
even  think  of  them  until  you  are  actually  involved  in  building  the 
system.  If  you  tried  to  specify  all  of  the  real-world  requirements 
and  the  regulation  interfaces  that  are  required,  your  RFP  would 
have  to  be  carried  in  a  wheelbarrow.  But,  those  are  the  things 
that  the  acquisition  agencies  and  the  contractors  need  to 
understand.  You  cannot  anticipate  everything  and  put  it  in  the  RFP 
and  SOW,  so  you  have  to  have  work  around  and  be  flexible  enough  to 
deal  with  the  real-world  constraints. 

Schedule  Versus  Quality 

It  really  made  me  feel  good  when  ASD  came  down  and  said, 
"Schedule  vs.  quality  is  not  an  issue;  we  must  always  have 
quality."  It  has  made  my  job  a  lot  easier  knowing  that  I  have  that 
support.  We  have  had  a  lot  of  schedule  perturbations  because  ol 
underscoping  and  so  forth,  but  whenever  it  came  to  quality  we 
slipped  the  schedule.  As  long  as  we  had  the  manning  to  keep  going 
under  current  courses,  which  we  have,  it  has  not  been  a  real 
problem  for  us.  We  have  had  to  study  it  closely,  but  it  is 
something  we  have  been  able  to  live  with.  In  my  view,  slipping 
this  program  for  four  months  to  get  the  quality  that  v;e  needed  is 
not  even  close  to  the  issue  of  having  to  live  with  a  program  that 
does  not  work  and  trying  to  fix  it  for  the  next  20  years.  So, 
while  we  are  in  acquisition,  let's  do  it  right.  If  you  meet  ttu- 
schedule,  that's  great,  but  if  you  do  not,  when  the  two  butt  heads, 
the  schedule  has  to  lose;  it  has  to  lose  if  it  means  sacrificing 
qual ity . 

1 ntegration!  Integration!  Integration! 

Remember  that  when  you  go  out  to  buy  a  house,  the  three  most 
important  considerations  are:  location,  location,  location.  On 

tlie  other  hand,  when  managing  the  development  of  an  ATS,  the  thico 
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most  important  considerations  are;  integration,  integration, 
integration.  Even  at  the  level  of  writing  a  lesson,  if  a  flight 
engineer  is  developing  a  simulator  lesson,  he  must  go  over  and 
integrate  that  lesson  with  tne  pilots.  Because  the  flight 
engineers  cannot  do  their  thing  in  the  simulator  independent  of 
what  the  pilots  are  doing,  the  lessons  have  to  be  integrated. 
Thus,  you  have  to  integrate  at  the  lesson  level.  You  also  have  to 
integrate  at  the  unit  level  and  at  the  course  level.  You  also  have 
to  integrate  across  courses  and  crew  positions.  It  just  keeps 
growing.  You  have  to  integrate  ISO  products  with  the  TMS 
capabilities  to  be  sure  that  they  will  work  together,  and  it  goes 
on  and  on. 

My  primary  job  when  someone  brings  me  a  problem  is  to  scope  it 
ana  determine  the  ramifications  for  other  parts  of  the  system.  For 
example,  if  someone  identifies  a  problem  with  the  TMS,  my  concern 
is  with  what  is  the  impact  on  courseware?  What  is  the  impact  on 
the  instructors?  What  does  it  do  to  simulators?  What  does  it  do 
to  the  schedule?  You  have  to  think  system-wide  every  time  you  face 
a  problem.  And  if  you  don't,  you  fix  one  problem  and  it  looks 
great  for  this  part  of  the  system,  but  it  may  have  a  negative 
impact  on  another  part  of  the  system  and  may  have  to  be  redone 
again . 


Integration  is  the  key.  When  the  contractor  brought  in  their 
two  integration  managers,  they  brought  a  lot  to  the  game.  They 
helped  accomplish  the  required  integration,  particularly  between 
TMS  and  ISO.  Integration  is  what  makes  it  work  in  the  real  world. 
You  have  to  manage  it  very  closely  or  they  will  lose  their  shirts 
in  terms  of  dollars,  and  we  will  lose  ours  in  terms  of  quality. 
Having  someone  on  the  contractor's  team  and  the  government's  team 
who  has  seen  it  before  and  can  avoid  the  major  pitfalls  is 
extremely  important,  and  vital  to  success. 


III.  CONCLUDING  REMARKS 

As  noted  in  the  Introduction,  it  is  believed  that  the 
information  presented  in  this  paper  can  be  a  very  useful  source  of 
guidance  for  both  acquisition  and  using  agencies  involved  in  the 
development,  implementation  and/or  utilization  of  aircrew  training 
systems.  The  information  provided  by  Lt  Col  Dukes  is  particularly 
credible,  because  it  is  based  on  the  actual  experience  gained  from 
his  deep  personal  involvement  in  the  C-130  ATS  program  from 
conception  to  implementation.  As  a  consequence,  at  the  time  of 
this  interview,  he  was  in  a  position  to  assess  the  validity  of  some 
of  the  early  design  and  development  assumptions  as  well  as  to 
provide  an  Air  Force  user's  perspective  concerning  what  was  really 
required  to  make  the  prograrr.  work. 

The  authors  recognize  that  the  specific  requirements  of 
particular  ATSs  may  differ  because  of  differences  in  the  size. 
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scope  and/or  complexity  of  the  programs.  In  addition,  programs  for 
new  systems  such  as  the  C-17  and  ATF  do  not  have  access  to  much  of 
the  historical  training  and  operational  experience  that  is 
available  to  existing  systems,  such  as  the  C-130,  C-141,  and  C-5. 
Thus,  there  is  probably  always  a  need  for  "tailoring"  in  certain 
areas  to  meet  the  unique  requirements  of  any  given  program. 
Despite  these  differences,  the  authors  believe  that  much  of  the 
information  provided  is  sufficiently  general  to  warrant 
consideration  by  most  of  the  new  and  existing  programs  with  which 
they  are  familiar.  The  authors  have  included  both  a  set  of 
references  cited  in  this  paper  as  well  as  a  separate  bibliography 
of  additional  publications  which  may  be  of  interest  to  any  reader 
desiring  more  information  in  this  area. 
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GLOSSARY  OF  ACRONYMS 


AFCMC 

AFORMS 

AFOTEC 

ASD 

ATD 

ATS 

AWM 

CASS 

CBT 

CCA 

CDR 

CIQ 

CLIN 

CLS 

CMI 

CPT 

CRR 

CWG 

DCAS 

DCM 

DID 

DO 

DOD 

DOO 

DOT 

DR 

DT&E 

DWI 

ICP 
ID 
ISD 
I  TO 
IVD 

MAC 
MATS 
MS  SR 

OBE 

OT&E 

PDR 

PMM 

PTT 


Air  Force  Contract  Maintenance  Center 

Air  Force  Operations  Resources  Management  System 

Air  Force  Operational  Test  and  Evaluation  Center 

Aeronautical  System  Division 

Aircrew  Training  Device 

Aircrew  Training  System 

Awaiting  Maintenance 

Computer-Assisted  Sguadron  Scheduling 
Computer-Based  Training 
Course  Configuration  Audit 
Critical  Design  Review 
Copilot  Initial  Qualification 
Contract  Line  Item 
Contractor  Logistics  Support 
Computer-Managed  Instruction 
Cockpit  Procedure  Trainers 
Course  Readiness  Review 
Configuration  Working  Group 

Defense  Contract  Administration  Services 

Deputy  Commander  for  Maintenance 

Data  Item  Description 

Deputy  Commander  for  Operations 

Department  of  Defense 

Operations  Center 

Director  of  Training 

Deficiency  Report 

Development  Test  and  Evaluation 

Departmental  Work  Instruction 

Instructional  Change  Proposal 
Instructional  Development 
Instructional  Systems  Development 
Individual  Try-Out 
Interactive  Video  Disc 

Military  Airlift  Command 
Model  Aircrew  Training  System 
Media  Selection  Syllabus  Report 

Overtaken  By  Events 
Operational  Test  and  Evaluation 

Preliminary  Design  Review 
Performance  Measurement  Module 
Part  Task  Trainer 
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QA 

QAPP 

QAR 

QC 

QDI 

QDR 

RFP 

SCNS 

SGTO 

SIMCERT 

SME 

SMM 

SNS 

SOF 

SOW 

SRB 

T&E 

TMS 

TO 

TQM 

TSR 

TSRR 

TSSC 

TTTS 

UTO 

WST 


Quality  Assurance 
Quality  Assurance  Program  Plan 
Quality  Assurance  Representative 
Quality  Control 

Quality  Departmental  Instruction 
Quality  Deficiency  Report 

Request  for  Proposal 

Self-Contained  Navigation  System 
Small  Group  Try-Out 
Simulator  Certification 
Subject  Matter  Expert 
Scheduling  Management  Module 
Satellite  Navigation  Station 
Special  Operations  Forces 
Statement  of  Work 
System  Review  Board 

Test  and  Evaluation 
Training  Management  System 
Technical  Order 
Total  Quality  Management 
Training  System  Review 
Training  System  Readiness  Review 
Training  System  Support  Center 
Tanker  Transport  Training  System 

Unit  Try-Out 

Weapon  System  Trainer 
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